this post was submitted on 20 Aug 2023

11 points (92.3% liked)

Self Hosted - Self-hosting your services.

14673 readers

2 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules

No harassment
crossposts from c/Open Source & c/docker & related may be allowed, depending on context
Video Promoting is allowed if is within the topic.
No spamming.
Stay friendly.
Follow the lemmy.ml instance rules.
Tag your post. (Read under)

Important

Lemmy doesn't have tags yet, so mark it with [Question], [Help], [Project], [Other], [Promoting] or other you may think is appropriate. This is strongly encouraged!

Cross-posting

!everything_git@lemmy.ml is allowed!
!docker@lemmy.ml is allowed!
!portainer@lemmy.ml is allowed!
!fediverse@lemmy.ml is allowed if topic has to do with selfhosting.
!selfhosted@lemmy.ml is allowed!

If you see a rule-breaker please DM the mods!

founded 4 years ago

MODERATORS

dogmuffins@lemmy.ml

testman@lemmy.ml

Zoe8338@lemmy.ml

Automatic updates: a cautionary tale (tarneo.fr)

submitted 2 years ago* (last edited 2 years ago) by tarneo@lemmy.ml to c/selfhost@lemmy.ml

14 comments fedilink hide all child comments

Tl;dr: Automatic updates on my home server caused 8 hours of downtime of all of renn.es' docker services including email and public websites

all 18 comments

sorted by: hot top controversial new old

[–] Moonrise2473@feddit.it 7 points 2 years ago (1 children)

I don't want to seem rude, but in my opinion automated unattended updates on Gentoo is a bad idea.

[–] tarneo@lemmy.ml 3 points 2 years ago* (last edited 2 years ago)

That's what I learned :-)

Edit: no saying that isn't rude

[+] yote_zip@pawb.social 3 points 2 years ago* (last edited 3 weeks ago) (2 children)

[deleted]

[–] tarneo@lemmy.ml 3 points 2 years ago (1 children)

I'm surprised this strategy was approved for a public server

The goal was to avoid getting hacked on a server that could have many vulnerable services (there are more than 20 services on there). When I set this up I was basically freaked out by the fact I hadn't updated mastodon more than a week after the last critical vulnerability in it was found (arbitrary code execution on the server). The quantity of affected users, compared to the impact it would have if hacked, made me choose the option of auto-updates back then, even if I now agree it wasn't clever (and I ended up shooting myself I'm the foot). These days I just do updates semi-regularly and I am subscribed to mailing lists like oss-security to know there's a vulnerability as early as possible. Plus I am not the only person in charge anymore.

[+] yote_zip@pawb.social 3 points 2 years ago* (last edited 3 weeks ago) (1 children)

[deleted]

[–] tarneo@lemmy.ml 2 points 2 years ago (1 children)

I totally agree. But I just wouldn't necessarily say gentoo is a bleeding edge distro: it's kinda up to the user. They are free to configure the package manager (portage) however they want and can even do updates manually. I just like the idea of having newer packages at the cost of stability, because I also use the server as a shell account host (with an isolated user ;-)) and need things like the latest neovim. These days I would know if an update failed because I would literally be in front of the process and test services are working after the updates, so I'd know if I have to rollback. This makes it basically like a stable distro IMO (even though the packages aren't battle tested before being pushed as updates).

[–] skilltheamps@feddit.de 2 points 2 years ago* (last edited 2 years ago) (1 children)

I don't know to what extent you got molested by the prophets of immutable distros yet, but I can only recommend to join the cult. Install Fedora IoT (or CoreOS) and simply know that you'll get a working container host (powered by podman) with every update. The whole discussion about which distro might survive whatever massacre the respective package manager commits next becomes superflous: You simply get the next image that was built upstream solely to serve containers. The whole package-udpating-shengiangs is done by other people for you, you only collect the sweet result. The only "downside" is that one has to become familiar with containers, but since you run docker already that should work out. Also for stuff like tinkering with the latest tools, just put those in a distrobox. That way they are indipendent from your solid container host, and you can mess them up in whatevery way you fancy and dispose them without any traces left behind.

Edit: To give one more example why this is awesome: It wouldn't even matter which one you install, you can just rebase to the other (IoT lives in the fedora-iot remote. silverblue, coreos and the others in the fedora remote. Just for anybody who might be confused by only looking at ostree remote refs fedora)

[–] tarneo@lemmy.ml 1 points 2 years ago (1 children)

To me, this is only one of the few advantages of immutability. I have already used nixOS on a server and I really didn't like having to learn how to do everything the right way. As for distrobox, to me it sounds quite like an additional failure point: it is an abstraction over the containers concept that hides the actual way it is done from you. I'd say if you run an app in a container, go all the way: make the container yourself. To me it just sounds like a bad idea, and I didn't really like distrobox when I tried it. I just want to say that both of these concepts (immutability, distrobox) would be great if it was perfectly done. But the learning curve of nixos and the wackiness of distrobox drove me away.

[–] skilltheamps@feddit.de 1 points 2 years ago* (last edited 2 years ago) (1 children)

The learning curve of NixOS is also what keeps me from trying it out, hence I prefer the "take it or leave it" mantra of the immutable fedoras, and try to keep the amount of packages I have rpm-ostree layer on top minimal.

As for Distrobox, yes there's ways it can fail, altough that happened rarely to me. What happens mostly is that the distro inside distrobox goes kaput because that's just what mutable distros beared with a plethora of questionable tooling installed with "curl something | bash" does. But for me that's the point of distrobox: separate all that shady cruft one may need for work/developing/etc from the host os. It's a place for messing about without messing up the computer and with it the bits that need to keep working

[–] tarneo@lemmy.ml 1 points 2 years ago

You convinced me for immutable fedora. Maybe I'll try it out sometime on our backup/testing server and maybe it will make its way to production if I'm happy with it.

As for distrobox I'll see.

The main reason I used Gentoo is because of being able to reduce the attack surface with USE flags. But as it seems the tradeoffs with it are greater than the advantages (the mastodon issue I mentioned). If I don't switch the server to immutable fedora, I'll just use something like plain fedora or debian I think.

[–] skullgiver@popplesburger.hilciferous.nl 2 points 2 years ago* (last edited 2 years ago) (1 children)

[This comment has been deleted by an automated system]

[–] thisisawayoflife@lemmy.world 2 points 2 years ago (1 children)

What is the reason to shy away from Ubuntu? It is pretty solid in terms of automatic updating and rebooting. I used to be hardcore centos but I gave up after all of the hubbub around 8. I just need to server to update, reboot when necessary and keep running all my stuff so I don't have to touch it. In my old age, I don't care to tinker anymore - I just want my services running and I want reports given to me about health and status.

Also, if you're concerned about privilege escalation, running a MAC is probably a good idea. SELinux saved my hide one a dozen years ago with a php bug where I did not sandbox an app properly. Thankfully, SELinux caught this and prevented anything bad from happening.

[–] tarneo@lemmy.ml 1 points 2 years ago (1 children)

what is the reason you shy away from ubuntu? Canonical. Snaps. Ubuntu is the first server OS I used, and while it was quite good I think I prefer using a base distrobox instead of a derivative. If I'm going to use Debian, I'll use Debian. Not Debian with corporate stuff on top.

As for SELinux: I've tried around a year ago. But as soon as I started doing stuff with users and tweaking docker permissions things went wrong and I just set it to permissive. Maybe I'll try that again soon, because other parts of managing servers have become much easier over time as I learned. I agree that having a server without SELinux is quite dumb and not very professional.

[–] thisisawayoflife@lemmy.world 1 points 2 years ago

Permissive mode is definitely a life saver. My path was usually exercising the application in permissive mode for a few days then running the SELinux scanner on the log file to determine what roles needed to be setup. Same with the Debian/Ubuntu equivalent.

Good luck!

[–] ReversalHatchery@beehaw.org 0 points 2 years ago* (last edited 2 years ago)

While we are here: what do you think about unattended updates on Debian and such? (as such being derivatives, including Proxmox VE)