I'll keep this short and to the point. If you aren't actively challenging the problematic culture, you are complicit in it. "I didn't do anything" is actually a huge part of the problem, because you are bystander allowing a violent rape culture to continue unimpeded. It's irrelevant how many perpetrators there are. What matters is the high number of victims - and all the other people who allow that dynamic to be normalized.
My vision of male privilege comes out of extensive documentation of just that by experts who have researched it. Again, the fact that you don't grasp this is because you aren't actually doing the work. Read Entitled by Kate Manne. Read Invisible Women by Caroline Criado-Perez. That doesn't mean that men have no problems, but one of the definitions of privilege is that it's so ubiquitous that you aren't even aware of it.
You very clearly aren't doing everything you possibly can. Almost no men are - or we wouldn't still live in violent rape cultures where women who speak up are issued death threats and told they are the actual problem. Despite that, women (and the men who are doing this work to shift the culture) are going to keep talking about it until things improve. If you're tired of hearing about how norms and behaviors that come out of mainstream masculinity are harmful, then join the people who are working to change them and stop blaming this shit on the women who are oppressed and victimized by it. We don't live in the culture that we do because most women like and choose abusive men. You can be confident and not be an asshole.
On a personal level, men who are not abusive toward women nonetheless play important roles in the lives of men who are. Men who physically and sexually abuse women are not monsters who live apart from the civilized world. They are in our families and friendship circles. They are our fathers, sons, brothers, and best friends. They are our fishing partners, drinking buddies, teammates, fraternity brothers, and colleagues. We too easily let them and ourselves off the hook when we call their violence a women’s issue. Do we do it intentionally? I don’t know. But whether conscious or unconscious, it’s an effective strategy to avoid accountability.
Katz, Jackson. The Macho Paradox (p. 17). Sourcebooks. Kindle Edition.
