While I disagree with your gender-oriented spin on this completely, and I don’t think it’s nearly as common as the picture you paint (never heard anyone roasted for liking Fight Club), it does occasionally happen.
I watched the first few episodes of MLP: FIM at a friend’s house by happenstance, and within about a week of airing I told a few people “Surprisingly, it’s actually a pretty good show.” Then the controversial fans came out and I completely stopped talking about the show to protect myself. Literal years later I found out my mom had been loudly proclaiming to anyone who would listen, including distant family, that I was a “brony”. It soured a few tenuous relationships with people who didn’t know me well enough to know it was an absurd label.
I really love Supraland, but it’s hard to convince people to try it for some reason.