We have a very highly individualized culture, so it's tough for many people to think about societal dynamics. They think, "If I didn't do it, then you don't get to lump me in with people who do." Nevermind that most racism, sexism, homophobia, etc., is implicit bias that comes not from overt hatred but from stereotypes and cultural narratives that we've been exposed to from birth and that often run us from behind the scenes, even sometimes when we have conscious beliefs that are different from them.
Cognitive scientists tell us that only about 2% of thought is conscious, but if you are deeply bought into stories about personal responsibility in a way that refuses to acknowledge how you help to co-create the society you live in, and believe that you direct your own life, then they are going to balk. Ironically, it's also a defense mechanism to not have to actually take responsibility for the society that they live in. It's a literal evasion of self-responsibility.
