I don't know where you got that from that I am implying anything of the sort. Patriarchy is a dominance based hierarchy where men have primacy, but the male part is only one part of entire social stratification. It's a pyramid of power where only a small number of elites benefit at the expense of everyone else and those "below" are always trying to step on each other to get up a higher rung and closer to the apex. White women who voted for Trump is one example. Black men who disrespect and marginalize Black women is another. It drives police brutality and schoolyard bullying.

That being said, the way that "traditional" masculinity is constructed drives and is aligned with that dynamic of always jousting for power and place at other’s expense, often using violence or bullying as a weapon. It teaches men to be stoic, in control at all times, to dominate others and that they have the right to control women. The most violence that is done to women is by men in their close personal sphere - their fathers, their lovers and husbands. It's what justifies and maintains rape culture.

Women uphold patriarchy too, and they have to do their part in order to allow masculine norms to shift into ones that are less violent and less isolating for men. But men have to lead the way on that because as we've seen for the past 50 years, most men are highly resistant to be told that stuff needs to change by women. The men who decline to challenge the existing culture are essentially colluding with it. The problem isn't men, it's masculine norms and gender indoctrination as it stands. You are not "at fault" for being born a man, but you are responsible for helping to co-create a better society. That doesn't let anyone else off the hook to behave decently, kindly, and with equality in mind.

Edit: the TL:DR is this
The idea that some people are not as worthy, as intelligent, as equal as others is rooted in the belief that women are not as worthy, as intelligent, as equal as men. Start shifting those beliefs via shifting how women are treated in this culture and it impacts all equality for the better.