Of course, we should care about everyone in the society, but expecting people who have been deeply disenfranchised by the social system - even the richest Black folks have experienced lifelong racism - and told that they exist only to serve the top of the pyramid to be the main champions of something other than their personal painful experiences is misogyny and misnoir in action. It’s saying, “You need to put yourself last in order to serve my needs.”
"Her humanity may hence be held to be owed to other human beings, and her value contingent on her giving moral goods to them: life, love, pleasure, nurture, sustenance, and comfort, being some such. This helps to explain why she is often understood perfectly well to have a mind of her own, yet punished in brutal and inhumane ways when that mind appears to be oriented to the wrong things, in the wrong ways, to the wrong people — including herself and other women."
Manne, Kate. Down Girl (pp. 22–23). Oxford University Press. Kindle Edition.
Although I would love to see a more egalitarian structure where everyone is cared about and their needs are considered to be for the good of all, asking people who have been discriminated against in a dominance hierarchy system for no other reason than what demographic they fit in to pretend that isn't a really relevant thing is also completely unrealistic. Part of what pains me as a woman is the ways that every other woman I know has had a lot of the same negative experiences that I have. That doesn't mean I don't care about what happens to anyone else - it means that the bulk of my advocacy is for people who have shared a similar plight. And it's not a contest - I seriously doubt that most people (other than a lot of white men I've seen) are trying to determine "who has it worst." They are simply trying to get other people to care about what's going on for them- particularly if they are in a demographic that has some power to change it.
If someone is bullying you - harming you both emotionally and physically - it would ideal to ask, "What is making this person behave this way? What pain or dysfunction in their own life is behind this?" But realistically, first you are going to try to get them to stop. Then maybe after you heal from your wounds, then you can begin to care about theirs.