In a truly polyamorous relationship, if your partner found someone else that he really cared about and wanted to spend time with, he wouldn’t have to leave you to do so. I think the first experience you had went so badly because it came as a “fix” rather than an enhancement to your relationship and your then husband just jumped into another monogamy construct. Rather than expanding his capacity for love and intimate contact, he just moved on to another partner.
Here’s what I wrote about this when asked about how to open up a marriage: “In other words, if your marriage is just a bit stale and there’s no longer much spark in your sex life, starting to see other partners isn’t really a fix. If anything, it will probably drive a deeper wedge into your relationship and take you off in separate directions. If you have a strong, healthy and sexually satisfying relationship and simply want to expand on that, then you have a chance of making this work, but changing relationship fundamentals mid-stream (rather than starting out by agreeing to open relationship) is challenging. It can work, however, if you approach it from the perspective of what it brings to you both as a couple. How would it enhance the relationship between the two of you, not just how can you essentially get a hall pass. If you don’t have a strong, healthy and sexually satisfying relationship, that is the place to put your attention first.”
Obviously, that being said, you should do what’s right for you and for your current relationship.