I thought Bowie's wife found him in bed with Jagger. At least that was the story I always heard as a kid. Anyhow, like you said, let people live how they wanna live, so long as no one is hurt.
I lived and went to school in Los Angeles and had some friends in wilder circles (like famous AVN starlet types...lol). It wasn't uncommon for guys who didn't "use" to be "holding" for when girls came to party.