So today I discovered that there's a cron job that holds non-reproducible state that died, and now our system is fucked.
The cron job doesn't live inside any source control. This morning it entered a terminal state, and because it overwrites its state there's no way to revert it.
I'm currently waiting for the database rollback and have rewritten it in a reproducible/idempotent way.
It's not horrible, like it'll do the job just fine, it's just probably a better idea to use systemd and like, containers and whatnot, but I couldn't be arsed to fiddle with all that for Jellyfin, caddy reverse proxy, and two modded Minecraft servers, so shell scripts and tmux won the day. It takes a little extra time to restart everything after an update, and maybe I'll get the motivation to do things "correctly™" one day, but today is not that day.
Use the tmux resurrect plugin. It will restore your tmux session to its previous state after a restart, including programs if you like.
You can put off doing things "correctly™" even longer.
The "here's how you keep doing this poorly but more efficiently" energy on display here is a refreshing change of pace from the usual "here's how you do this correctly" crap peddled by normies (including me). You have my respect.