I’m looking for some feedback on my Plex system architecture.
All my media is stored on a Synology DS 1621+, six 4 TB drives in RAID 6 with one acting as a hot spare. All four network ports are bonded into a 4G link to an Ubiquiti USW-48-POE.
Previously, I ran Plex in a Docker container on the NAS. This setup was stable; however, the NAS only has 4 GB of memory shared between Plex, several other Docker services, and regular DSM overhead. Plus, the processor is not very powerful (AMD Ryzen V1500B, ~5400 PassMark).
A few months ago I repurposed some old desktop PC parts to build a home lab Proxmox server (Core i7-6700K [~8900 PassMark], 32 GB memory, GTX 970, an old 2.5” SATA SSD for guest OS disks, 1G networking on the motherboard). I’m running Plex on an Ubuntu VM, with the GPU passed through directly to the guest OS. Plex is not containerized in Ubuntu. The VM has 8 CPU cores and 8 GiB memory (different units in Proxmox). My Plex media is accessed via a persistent NFS mount in Ubuntu (had been SMB before a DSM update broke something and the VM could no longer read the directory contents.)
The main purpose of the change from NAS to VM was to utilize the increased CPU/GPU horsepower and memory that I had lying around, but I worry that the added layers of complexity (hypervisor/VM, PCIe pass through, NFS mounts) will introduce more opportunities for performance issues. I have noticed more frequent hiccups/buffering/transcoding since the change but I’m not sure if it’s related to my setup or if those issues lie with client devices and/or the files themselves (e.g. weird file container type that the client can’t play natively).
Any critique or recommendations on system architecture? Should I get a dedicated NIC to pass through to my VM? Dedicated NVMe drive passed through as a guest OS disk? Ditch Proxmox altogether and go back to Synology Docker container?
I'm kind of over the whole idea of keeping "pets" around to serve my various self-hosting needs. Why make a hypervisor and then shard off pieces of this and create multiple operating systems that need to be maintained when you can just orchestrate all your hosting needs with a container orchestrator like k0s/k3s on the host? Even GPU passthrough can be done.
I'm a bit biased because I'm also a CKA, but I was a die-hard "bare metal or bust" kinda person with my self hosted stuff until I discovered kubernetes. K8s is a lot of resources on its own but a distro like k0s really pares down on the minimum requirements and basically just becomes a more featureful version of Docker if you just run it as a single node.
Eventually I came to understand that when your entire home stack is represented by a few hundred lines of yaml and a couple directories of portable data, that you can stop coddling the Linux install and just use the applications.
I apologize for my ignorance when it comes to Kubernetes - I sort of wrote it off as complete overkill for a home lab when my very basic understanding was that it was essentially a load balancer. After some light research, I'm beginning to understand that it could be a better solution than a full-blown hypervisor.
If I understand your comment correctly, you're suggesting to simply run a lightweight distro and install k0s or k3s to run containers? What would be an ideal bare metal OS for this? What would be pros/cons to k0s vs k3s in a home lab environment, or is that simply a matter of personal preference? What would be the best way to connect to my media - SMB, NFS, something else? Or are the differences here irrelevant? Any concerns (permissions, IO latency) when passing an NFS mount from host into a container, or is there an even better way to do something like that entirely within the container?
A complete kubernetes cluster for a homelab probably would be overkill (unless you really wanted a kubernetes playground, which some folks do). However, yes, my recommendation these days would be k0s directly. I did use k3s up until recently but gave k0s a shot and realized it's a bit lighter on resources, more configurable (for instance you can choose to run cri-o instead of containerd, which isn't an option with k3s) and has some extra features like letting you put helm charts with their values directly in the k0s config.
k0s vs k3s just comes down to personal preference but for me it came down to:
As for distro to run it on, I use MicroOS myself (immutable os, I have it set to automatically update and reboot once a week), but Debian is my second choice and my personal preference for server distros. The beauty of this setup is the container host really just needs the bare minimum to run the containers. There's less that can break because the containers are all managed by others upstream so the main concern of breakage areas basically becomes did server boot and did k0s start?
NFS is fine and actually natively supported as a kubernetes volume type: https://kubernetes.io/docs/concepts/storage/volumes/
Your option could be to mount it directly to the host first and then use a hostPath to mount it to the container, or just mount the NFS path directly to the pod. As for permissions you may need to do some mapping, but kubernetes also has security contexts that can let you alter the UID of the user running the pod. If you need user to be privileged and root, you can do that or if you need UID 5124 you can do that too.
If your goal right now is a Plex server and not much else to start then this makes things very easy:
You can add nginx ingress, cert-manager, metal lb, etc later on down the line if you get curious and want to expand a bit (sonarr, radarr, adguard home, etc)
You could also just go full stupid with kubevirt, but it's not a project I've personally explored using. Iirc it basically allows for the provisioning of more persistent VMs with k8s rather than containers
Again, pardon my ignorance when it comes to Kubernetes. Why would I use something like k0s instead of just regular old Docker? I suspect PCIe passthrough will have similar challenges on both k0s and Docker, whereas on Proxmox it's been relatively painless.
This might be better suited for a different community, in which case I'll make a post where appropriate. I'm not familiar with some of the Kubernetes terminology - batteries, pod/manifest (is this similar to stacks/docker compose?), NodePort?
I mean you could just use docker and if all you want is a plex container, it may be the way to go. Kubernetes is definitely a lot to learn if you are kind of hesitant to get started with it in the first place. I would just say that single binary distribution like k0s just basically becomes a docker on steroids when used in a single node environment. I've become so familiar with k8s that going back to docker feels like a massive downgrade for anything but a simple and straightforward task (which a single plex container, admittedly is).
Just do yourself a favour and if you go that route, at least use docker-compose to template your container. Searching through your bash history to find the command you used to start the container is just a recipe for frustration waiting to happen.
One major feature that k8s has which docker doesn't (I mean, there are lots tbh) is the ability to use helm charts. These are basically install templates, if you don't like the defaults then you can provide your own (assuming the chart author has written good charts) and the helm chart will template your values into the default chart and spin up a bespoke version for you.
For instance, a theoretical helm chart whose purpose is to install qbittorrent would likely provide the following:
However, say this ingress by default doesn't use SSL, but you want to ensure you're using https when you enter the password on your web interface, but the rest of the chart defaults meet your needs -- then a well written chart would allow you to provide values that let you template the SSL setup for the ingress object and provide for cert-manager to go and provision some certificates for your provided hostname with Let's Encrypt.
Helm charts basically are a way to provide a sane set of defaults which can be extended and customized to personal needs.
k8s (or the lightweight cousins) may not be ideal for you and, as I said, I'm biased because I'm a certified k8s admin, so tinkering with k8s resembles something like fun for me. Your mileage may vary :)
Terms: