For most people, the lying and dishonesty that accompany not being faithful are a huge breach of trust and are absolutely a big deal. As a polyamorous person, I agree that your partner having sex with someone else doesn't need to be a big deal - but if it's not being done openly and honestly, then it absolutely is.