Funny

This sentence from the Wikipedia article makes a lot of sense to me:

The style was further popularised in a 1955 essay by architectural critic Reyner Banham, who also associated the movement with the French phrases béton brut ("raw concrete") and art brut ("raw art").

So, the adjective wasn't "brutal", but "brut" meaning raw. The buildings were supposed to be raw. Instead of decorations in the stone like gargoyles, or even basic things like a decorative frieze, they'd just leave the raw lines required to make the structure sound. From that point of view, I get it. Reduce all the decoration to a minimum and let the structure "speak" for itself. Brutalist architects weren't intentionally making buildings ugly and "brutal". They were trying to make them clean, simple and undecorated.

A brand new brutalist building that's dry and unweathered on a day with bright sunlight and a blue sky might look nice. It would showcase the architect's design of the building, rather than some other artist's design of a gargoyle or other decorative feature. But, like in the picture, a wet, weathered building on a grey, overcast day is different. There's not much contrast between the building and the sky. The clean, monochrome concrete looks weathered much more quickly than natural stone. Also, it just looks like it's function over form, which is something we associate with places that aren't built for the public to enjoy: warehouses, military structures, even prisons.