I've been thinking about this for several hours since I first became aware of the debate.
I don't care that much in theory if anyone sees my votes. They aren't anything I'm particularly private about. I care about conversation way more than up/down votes.
However, some people get a little upset about being downvoted. I think it will result in retaliatory downvotes. You already see that when two folks are arguing. I don't normally waste my time downvoting a post I'm writing a rebuttal to, but when they are downvoting me I tend to do it back. I think if everyone had easy access, they would hunt down their down voters posts and retaliate regardless of the quality of the comments.
Lastly, I wonder if this will give rise to a client that lets you use one account to post/comment and a different one to vote. And if it does, will that be better all around? Then no one will be able to associate votes with a user. But it seems unnecessarily wasteful to create a whole account that does nothing but vote. It seems like it would deny mods (and everyone) a useful tool for identifying bad actors.
Technically, anyone could get access to the voters identity if they try hard enough but 99% of the users won't put in that much effort. And technically someone could already use different accounts for different activities, but without reason to create a client to support that it's too much of a pain to be worth the effort.
So I really think I'm on team status quo here.