If one is trying to "help" women and is "on their side", and they are constantly upset with you about both your attitudes and your words, Occam's razor says that you are the actual problem. Whining "Not all men" doesn't indicate actual allyship. Neither does pretending that patriarchy doesn’t exist. It says, "I hope things can get better for you, but not if it inconveniences me or makes me have to take a look at where I might be contributing to your problems as a function of living in this social system and how I’ve been taught to think and act by it."

The DARVO coming from both you and him is why we can't have nice things.