Hi all,
First off, I want to apologize for all the server instability. We long ago outgrew our instance size, but I was unable to afford a larger node on our provider, Vultr. We were maxing out every part of the server whenever any even slightly significant number of users were on the fediverse.
I've finally found the time to migrate us to a new provider, which allows us to step up to a much more powerful configuration. That migration has now been completed. I actually intended to post about the downtime on this community this morning before beginning, but when I went to do so, the server was already down and struggling to come back up. So I went ahead with the migration.
Server before 4cpu/16GB/400GB NVMe
Server after 8cpu/64GB/1Tb NVMe
Please update this thread if you are seeing any issues around any part of the site. This means duplicate threads, things that aren't federating, inability to load profiles, etc.
There is still database tuning that needs to occur, so you should expect some downtime here and there, but otherwise the instance should be much more stable from now on.
During this process I also improved several other aspects of operating the server, so any 'actual' downtime should be accompanied by proper maintenance pages (that hopefully don't get wiped by ansible anymore), so that will also be a good indicator of legitimate maintenance.
Once again, I really apologize for all of the downtime. It's very frustrating to use a server that operates like this, I understand.
snowe
Thank you for all the work that goes into maintaining this instance!
and thank you for being a great contributor to the community! this site would be nothing without all of you!