And what the people who use that phrase conveniently forget is that according to Darwin, fitness has nothing to do with strength or aggression - those who are the most adaptable are the ones who have the fitness to survive.
As noted primatologist Frans deWaal has said, " “I am not trying to ignore the role of aggression and competition in understanding primate and human social interactions. My perspective, however, is that affiliation, cooperation, and social tolerance associated with long-term mutual benefits form the core of social group living.”
He also said, “Destabilization of the social resource network decreases group stability and efficiency and lowers the average fitness benefit derived from cooperation. When group stability is important for individual advantage, selection will favor active peacemaking and cooperation in our closest relatives and ourselves.”
The first evidence of war is from 13,000 years ago and most of it is from 8k years ago or later. If you have low population density, ample natural resources, a lot of kin in neighboring bands, and more advantages accrued from cooperation than conflict, what is there to even fight about?
Four other hominid species did not survive - we did and a large part of that has to do with the ability to cooperate and share. We forget that at our peril.