Losing people who aren't a good fit for your boundaries/needs makes room for others who are a better fit. It's not fun to make that shift, but probably better off in the long run.