Hetzner is a good provider. Congrats on the migration and good luck.
this post was submitted on 15 Oct 2025
65 points (100.0% liked)
Programming.dev Meta
2691 readers
35 users here now
Welcome to the Programming.Dev meta community!
This is a community for discussing things about programming.dev itself. Things like announcements, site help posts, site questions, etc. are all welcome here.
Links
Credits
founded 2 years ago
MODERATORS
When programming.dev became practically unusable most of the time in the last several days I was considering moving elsewhere. No post or announcement even acknowledging the issue here in meta didn't make me hopeful in the issues subsiding at all. (I've experienced instance death before on feddit.de.)
Great to see this development.
Thank you for your continued work and efforts! ๐
Felt the same way, and when I had time and thoughts to post the server was down. Glad to see it back up with an upgrade.
Very happy to hear this. I was noticing frequent slowness recently. I never really got to the point of considering leaving because I'm not on here that much anyway. But I do sponsor your github in a small way. I hope the sponsorships cover at least the hosting cost.
You might want to update your Github sponsor page as it sounds like some of that info is out of date after this migration.
it is helping, thank you for the sponsorship. I should have migrated a long time ago because the costs really were adding up. I'll update my sponsor page after I have a fresh month of data for the bucket costs (which are still on Vultr) and the new server costs (which hopefully should be static). Thanks for the suggestion!
Great!
Thank you for all the work that goes into maintaining this instance!
and thank you for being a great contributor to the community! this site would be nothing without all of you!
Thanks, boss. You are a good man. ๐บ
Out of curiosity what provider did you guys move to that gives better bang for the buck than vultr?
Hetzner. Honestly every provider was cheaper. I literally didn't find a single provider that was even close to as expensive as Vultr. You can look at Vultr's deploy page here (might need to be logged in for that). For 16GB of RAM on any product, the minimum cost is $80 a month. We were paying $120+.
It's honestly crazy how expensive Vultr is. The servers might have better processors, didn't really check that, but all our performance depends on RAM and cores, so none of that really matters.
Also was able to get 64GB of ECC RAM on Hetzner. No clue if Vultr provided that, but they don't list it anywhere.
Providers I looked at:
- Scaleway
- Hetzner
- Contabo
- Netcup
- OVH
- Space Hosting
- I think one more, but can't remember it right now.
Thanks, it seems a lot faster!
Fantastic, thanks! I might be spending more time on Lemmy after all.
Oh, cool, to be honest I was actually in the process of transitioning to another instance due to the instability, but it sounds like I may not need to!
Thanks a lot!
sorry for all the trouble. I would understand it if you still do, as I haven't been the best operator.
Good news. Thank you.
And it's good to hear that it wasn't something nefarious messing with the instance.
Oh we are getting attacked constantly, that definitely didn't help, but a large portion of it was just thrashing from postgres not getting enough memory.
Sidebar stats say this instance gets 23 users/day, which seems absolutely tiny and within the capabilities of 4c/16gb cloud instance.
We were maxing out every part of the server whenever any even slightly significant number of users were on the fediverse.
Idk anything about how lemmy/fediverse works, but does that mean tiny instances like this get hit when the rest of the network is experiencing high load? Seems problematic.
EDIT: btw, thanks for the free service and the effort you put in to keep it running!
programming.dev is the 9th largest lemmy server. https://join-lemmy.org/instances
That stat was probably that low due to the server being down for around 90% of the last two weeks. If you look now it's at 220 and it will continue to go up.
On top of that, every action on every server that is federated is relayed to every instance. So all of lemmy.world's activity is still relayed to us and we have to handle it. Same for the other servers.
On top of that we also operate many other services:
- bytes.programming.dev
- git.programming.dev
- blocks.programming.dev
- etc (there's a lot)
But really it was mostly just postgres thrashing on all the requests. Here's a look at our Cloudflare dashboard for number of requests:
Yes this should be handle-able by a server that small (think actor paradigm), but I was unable to tune postgres to get it to that point as I'm not great at database stuff. I'm sure a DBA would have done a better job. I will note that some of the queries being used in the lemmy code are very badly optimized and were taking 20+ seconds to run each time, locking up the instance. With that on top of some other badly optimized selects for things like reading comments (which would take like 7s mean), there wasn't much I could do.
With the cost difference it was well worth it to just upgrade to a cheaper better server all around.
For all of the attention in the early days about Lemmy being rust based and thus focused on performance, the database seems to be the main bottleneck neck and from anecdotal monitoring of the other admins complaints I'd say that seems true.
Seems like some design issues lead to heavy database usage and its going to be really hard to optimize away from that.
I don't really have a better idea, just acknowledging even a small instance has to scale disproportionally to its size when the rest of the network grows and that's heavy on the database specifically.
The push-based ActivityPub (apub) federation itself is bad design anyway. Something pull-based with aggregation and well-defined synchronisation would have been much better.
There are ideas beyond that. For example, complete separation between content and moderation. But that would diverge from the decentralized family of protocols apub belongs to, and may not attract a lot of users and traffic. And those who care and don't mind smaller networks prefer fully distributed solutions anyway.