Because generally social interaction is easier and better face to face. You can read people's facial cues better, have true eye contact, better hear the subtitles of voice and mood. People feel more connected with someone if they have met them face to face.
Alternatively, communication via email and video call can be hard and easily misread. People can misread emails as aggressive or be aggressive and not realise the impact. Communication on a video call, especially in big groups, can be difficult and impersonal.
Meeting up occasionally is probably seen as good a way to keep your team coherent and friendly. You're more likely to be aware of the other person's feelings if it's someone you've socialised and spent time with. It's easier to be empathetic and kind if you know that person in the flesh rather than just a name on an email or a random face on a video call. You're more likely to make allowances for other people if you know about them and their circumstances.
When working remotely how many times do you have social calls and chats with your colleagues? It's an important element of being in a long term team.
I work in a hospital in a busy face to face job but some colleagues I barely see as we have different weekly rosters. So I only interact with them via email or video call; despite being in the same building a lot of the time. We make the time once a month to have a team meeting and social catch up as it's good for everyone and the team. It's similar to what you're doing once a year across a country.
You may not see the value in it but it may be worth noting other people may see the value in getting to know you and understand you. For example if that socialisation isn't something that comes natural to you, your team members seeing you and getting to know you will also help them adjust to work better with you. It is very much a 2 way thing.