None of us live on an island. We are all inextricably intertwined in our culture(s). There is nothing related to a society that can only be about one thing - and patriarchy sure as hell is not just about the disenfranchisement of women. Feminism opposes patriarchy, primarily for the benefit of women, but dismantling systems of hierarchy enforced through domination benefits everyone. Getting rid of tiny little boxes of acceptable gender expression benefits everyone.

Men benefit from the oppression and marginalization of women whether they want to or not, just as dominant races benefit from racism whether they want to or not. There is no "opting out" because it's not about you. Most of what is termed "women's issues" are actually "men's issues" - the attitudes and behaviors that men have been indoctrinated into that harm women, children, and other men. You are either a part of the problem - because you turn a blind eye, or perpetrate harm - or you are a part of the solution. There is no standing on the sidelines.