I'm polyamorous so I have no issue at all with multiple sex partners, but it sounds like this guy is involved in several relationships (even if primarily sexual relationships) and not just having "hit it and quit it" hook-ups. I do think there's a rather large difference.