"We disagree over how much to credit or blame men, for creating a "patriarchy" from allegedly egalitarian systems that preceded it. That seems to be a matter of perspective."
In other words, what you actually care about is not being blamed for all the harm that patriarchy still wreaks. That's the only thing that matters to you. But it's a social system, one that arose for complex reasons, not out of the malice of individual men. If you'd stop centralizing yourself in this story, you'd perhaps be able to appreciate that and start caring more about how it negatively affects us all, some to a much worse extent than others.