There's little evidence that there were ever “big time harem masters” until the last 3% or so of human history. We would have never succeeded as a species if there had been. Conversely, there's a huge amount to indicate that humans survived and evolved because they were cooperative breeders. Small bands of 20-50 helped each other out and took care of each other in all things, including sharing food and child-rearing. Partible paternity is still practiced in many places around the world, where a child has more than one father. There are also lots of other forms of non-monogamy taking place all across the globe even today in many cultures.
Sexual control of women (which would have been nearly impossible in a hunter-gatherer tribe) did not begin until the onset of patriarchy 6-9 K years ago. For the bulk of human history, social cooperation, including cooperative breeding, would have been a much more effective survival strategy.
Also, polygamy and polyamory are not remotely the same thing. Polygamy has an inherent power differential where the man is still “the leader” of the family/harem. Polyamory has no inherent power structures and has an ethos of individuality and egalitarianism even if not every single polyamorous unit abides by that ethos.
I've done a lot of research on this. I’m also happily polyamorous. Here are just a few of the stories that have come from it.
"What the latest science is telling us is that most of our ideas about the natural state of things, particularly as relates to female sexuality and sociology is really off base. Much of the current cultural narrative comes from what we’ve seen in movies about our ancient ancestors, and that, in turn, was either invented wholesale by Hollywood or was taken from incorrect assumptions made by earlier scientists who were steeped in patriarchal constructs."
“In a demographic simulation that Omkar Deshpande, Marcus Feldman and I conducted at Stanford University, California, we found that, rather than imparting advantages to the group, unequal access to resources is inherently destabilising and greatly raises the chance of group extinction in stable environments."
Polyandry was normal in pre-contact Polynesia, particularly for high caste women and still takes place in the Indian Himalayas and in parts of Tibet. In Lowland South America, and in Africa partible paternity, where two or more men mate with a woman for the purposes of producing a child, is common in many cultures. Spreading fatherly feelings throughout the group helps to maintain solidarity and cohesion as well as promotes the wellbeing of a greater number of children. Reproductive fitness (the chance that offspring will, in turn, produce their own offspring) is enhanced by cooperative alloparenting of this kind where several adults take an active interest in the lives of children.
“Despite the belief that monogamous male-female bonding is how mothers and children were supported and thrived, the anthropologist Sarah Blaffer Hrdy and others believe it was actually female cooperative breeding, or alloparenting — ‘sharing and caring derived from the pooled energy’ of a network of ‘grandparents, aunts, uncles, siblings, distantly related kin, and non-kin’ — that shaped our evolution.” (2)
