Why does what he does and who he is friends with affect you at all? This is all yours and not his. I’m not meaning that in a judgemental way, but just to say that you are buying into territorialism that is not necessary and seems to make you really unhappy. What would happen if you gave it up? Would your relationship change in any way, except perhaps for the better? I think you are causing unnecessary pain to yourself and to him due to social constructs that you are not obligated to buy into.