I dunno about harder, but this is absolutely a thing that is very rarely discussed.
I have a friend who is very conventionally attractive and the stories I hear about the way people treat them absolutely blows my mind. The out of pocket things people think they're entitled to say, from passersby on the street to medical professionals, the way they are literally predated upon, it's vile.
Sure, they get the benefits of being hot, it's a social lubricant for sure, but what they go through because of it sounds utterly exhausting.
They're wildly intelligent and absurdly hardworking to boot. But people don't take them as seriously as they should because all some people see is the flesh they're contained in. That particular issue affects both the conventionally attractive and the non-. Our society seems too obsessed with looks on the whole, to our collective detriment.
Oh, honey, it's not about protecting people...