My guidepost at this point in my life is to ask myself, "What is most loving to me?" If what other people want or need is wildly out of alignment with what is most loving to myself, I choose myself over them. This almost never happens with those I have warm reciprocal relationships with, but it is helpful in dealing with takers and narcissists. I finally learned (at almost 60) that I cannot wait for others to do what is most loving to me. I have to claim it for myself.
Wishing you all the best as you navigate this difficult situation. My personal opinion is that it's modeling healthy behavior to your children to create a lot of distance from those who don't respect your boundaries and needs. But, I know you'll figure out what works best for you in time.