It's definitely a deflection and a way to minimize your concerns, whatever the reasons may be. Too many men, and white men, in particular, have a willful ignorance of power dynamics or what it feels like to be systemically oppressed.