Edit: I've made an account here on lemmy.ml as I routinely can't comment or post from my account on lemmy.world.
Bit of a week! As usual, had a lot of fun tinkering. Here’s my takeaways from this past week(ish).
I finally learned how to set up a cron job with elevated privileges
This is something I've had on my , "I should really get this figured out" list for about two years now, but instead have been inconsistently typing my rsync commands (Since I've also been too lazy to set up the aliases for these commands).
I spent a couple of days rebuilding my server from the OS up (for reasons which I will explain momentarily), and since I'm up on a fresh OS with all my containers and services up and running, I figured it was time I figure out this cron job thing.
The approach I took was to write a simple bash script for my backup. The script is four lines. Three of which are sudo rsync ..., and the last of which is a curl -d ... command.
The rsync commands are to incrementally back up my server data, cache, and docker volumes.
The curl command triggers a notification through my ntfy instance, (link is to ntfy, not to my instance), to let me know the backups have successfully completed.
In order for that to run properly, I also had to learn....
How to update sudoers privileges
After reading about crontab and privileges, I know I could have just edited /etc/crontab and run my script as root, but what would be the fun in that when I could also learn about changing privileges through sudoers! So I learned how to modify sudo priviliges by creating a new file in sudoers.d with the command:
sudo visudo -f /etc/sudoers.d/name-of-my-sudoers-file
And why that and not just editing /etc/sudoers directly with nano or vim or emacs? That was my first question when I saw that command and thought, "Oh, shit, I'm going to have to brush up on my vi/m."
Turns out, if you just rando edit sudoers (or add a file to /etc/sudoers.d/) with any old editor, you can fuck up the syntax, and if you fuck up the syntax, you can fuck up your ability to use sudo, and then you can't do anything requiring sudo on your machine without going through tremendous headache to fix it.
However, if you use sudo visudo ..., you get syntax verification to prevent you from breaking sudo.
And, on Ubuntu server, visudo uses nano by default, which meant I didn't have to worry about vim just yet (vim is on my roadmap of things to learn)
(Also, you can change the default editor visudo uses, but I don't remember the command because I won't be changing it until I get a grip on vim and can make a decision about which editor I want to use.)
With all that being said, I created a file in /etc/sudoers.d and added a line to allow my backup script to run with elevated privileges without requiring a password with this syntax:
username ALL=(root) NOPASSWD: /path/to/my/script
Good documentation/notes will save you like good backups will save you
This isn’t something that’s new to me, or to linux (Arch wiki ftw) but it’s something that 100% made rebuilding my server from the OS up a pretty worry free breeze.
So why did I rebuild my server again a little over a month after rebuilding it from the OS up? Turns out when you accidentally kick a stool while carrying a heavy box and that stool knocks the fuck out of your server, your OS can get fucked up.
This happened a few weeks ago, and boy was I panicked when I first kicked that stool into the server. After putting down the boxes I turned on the monitor and the screen was freaking out. It looked like a scrambled Max Headroom. I held the power button to force a shutdown, and after rebooting the server everything came back up and I thought, ”Holy shit, I dodged a bullet!”
(Bonus lesson, I learned to not leave the stool in front of the server rack!)
But, all was not well. My server data and cache are on zfs pools, and every time I tried to bulk add some of the shows or movies I prepared to the data pool, I would get this procsys kernel panic error. I had repeatedly been checking my zpool status, and everything was good there. So I was furiously searching trying to figure out what the error meant, and I kept finding folks with the same or similar errors who talked about checking logs, but whenever I checked logs I couldn’t find anything to indicate what was actually going on.
Finally, after a few days additional searching, I ran across a comment on a thread that said this particular error (I neglected to save the error, I wish I had) was usually a hardware related issue, like a loose connector, and I thought, ”Holy shit, that makes perfect sense after knocking the shit of my server!”
So I shut it down, opened it up, and sure enough there was a loose cable on the motherboard. I reseated it, checked the rest, rebooted, and over the course of the next week, it seemed all was well.
But I kept getting these weird errors. Not actual error messages, no more kernel panics, and data wrote to the zpool just fine. It was little things not working as expected. Commands that typically ran very speedily (like ls) were lagging, opening a file in nano took multiple seconds instead of being near instant, stuff like that.
I decided to go for a nuke and pave approach, rebuilding from the OS up again, which is where the documentation comes in. Since I started messing about with self hosting 2-3 years ago, I’ve kept meticulous notes on everything I have done and learned so that if I had to re-do it, I could open up Joplin, search for whatever I needed, and proceed. This has saved my ass multiple times over the years as I tinker, break shit, and fix it using my notes.
So yeah, in addition to having a good backup system, you should also keep good documentation for yourself.
edit: removed extra 4 from post title
We're totally fucked.