It was probably just the Mormons that ruined that for you.
I'm sure in a different time line where the Mormons where not so controversial and vitial in their role of stealing land from mexico and the natives then it might not of been so much of a bad thing.
But I think most of our laws against poly stem from the the feud between the united States and the Mormons. There was some drama about redrawing boundary lines or the Mormons wanted to succeed and the state/feds made poly illegal as one of many ways they targeted the Mormons. I think being poly was precieved as slavery. Which under the Mormons it probably was essentially slavery to be fair.