No, that's just silly and reductive. More than one thing can be true at a time and in the case of privilege, often is. I have privilege as a white person, but also disadvantage and oppression as a woman. And this isn't feminist saying this - it's mainstream sociology and psychology.
Not that long ago the American Psychological Association issued new guidelines for working with men and boys to address all the ways that patriarchal norms of masculinity are destructive to males.
"Though men benefit from patriarchy, they are also impinged upon by patriarchy," says Ronald F. Levant, EdD, a professor emeritus of psychology at the University of Akron and co-editor of the APA volume "The Psychology of Men and Masculinities."
Thirteen years in the making, they draw on more than 40 years of research showing that traditional masculinity is psychologically harmful and that socializing boys to suppress their emotions causes damage that echoes both inwardly and outwardly.
The main thrust of the subsequent research is that traditional masculinity - marked by stoicism, competitiveness, dominance and aggression - is, on the whole, harmful. Men socialized in this way are less likely to engage in healthy behaviors." (apa.org)