I disagree. If you are in a relationship with someone, no matter how the terms are defined, you do have a say in things that affect your sexual health. If you’re having sex with multiple partners, you need to be aware of what you’re potentially being exposed to, and be honest with your other partners (like your husband) about what you might be exposing them to.
Having fuck buddies is great, they can be super satisfying relationships. But just because you’re not married or even in a formal dating relationship doesn’t mean that you shouldn’t have mutual respect and honesty with your sexual partner(s).