Those guys also have an outsized influence over what someone would have to do to be seen as virtuous. The guy being pelted with rotten vegetables and publicly executed for opposing them? Not seen as virtuous. The guy running a church that does some good things but in a way that reinforces their power? Seen as virtuous.

Even if people are overall good, that doesn't mean they can translate that into successfully coordinating to come to correct collective agreement about complicated problems and be immune to elaborate efforts to distort their beliefs in particular directions. If you think, here is what people are saying about what I should do, so that must be an accurate expression of their combined good intentions, well no, it isn't, because that isn't something that is easily achieved by default.