Fediverse

The data needs to be sent from the voter to the server that owns the post. But the server that owns the post can anonymize the data before it sends it to clients or other servers.

That said I don't have deep understanding of activitypub. Its possible that something would prevent this, like if votes made their way to the server that owns the post by way of telephone game rather than directly from the client or the user's home server. But that seems like an unreasonable design, so I doubt (hope not) that is the case.