So I have rebuilt my Production rack with very little in terms of an actual software plan.
I host mostly docker contained services (Forgejo, Ghost Blog, OpenWebUI, Outline) and I was previously hosting each one in their own Ubuntu Server VM on Proxmox thus defeating the purpose.
So I was going to run a VM on each of these Thinkcentres that worked as a Kubernetes Cluster and then ran everything on that. But that also feels silly since these PCs are already Clustered through Proxmox 9.
I was thinking about using LXC but part of the point of the Kubernetes cluster was to learn a new skill that might be useful in my career and I don't know how this will work with Cloudflared Tunnels which is my preferred means of exposing services to the internet.
I'm willing to take a class or follow a whole bunch of "how-to" videos, but I'm a little frazzled on my options. Any suggestions are welcome.
If you decide to go the Kubernetes route, you can try k3sup to bootstrap your VMs k3s, it a nice half step abstraction between Ansible and running curl yourself:
https://github.com/alexellis/k3sup
I've landed on k3s as my k8s distro in my environment for a number of reasons. It seems to have the "mindshare" of selfhosters, and theres lots of k3s documentation to peruse. I also really like that you can preload manifest files if you do decide to use Ansible, which makes cluster deploys that much more organized.
If you want to go a little off beat, you could try "Canonical K8s (not Microk8s)" as a snap. That worked REALLY well, and lets you do cool shit like "k8s enable loadbalancer" to automatically enable whole components for you, if you just want to focus on "consuming" Kubernetes instead of building it. I did notice a little overhead doing it as a snap, but my Proxmox node that runs the VM is purposely low spec (Celeron quad core if you believe it, 7 tdp tho)....so your hardware wouldn't likely notice a difference.
https://documentation.ubuntu.com/canonical-kubernetes/release-1.32/snap/tutorial/getting-started/
If youre doing Proxmox already, if you don't already have a VM template and/or Terraform/OpenTofu with Proxmox operator....it may help to tool on that too. Easier to destroy/build VMs when you get frustrated.
OP, If you dont have a proxmox vm template ready to go, here is a great starting place using cloudinit:
https://github.com/UntouchedWagons/Ubuntu-CloudInit-Docs
You can use this with the proxmox gui cloudinit config as well to add your ssh key to each vm/etc.
Not sure why but I had an absolutely horrible time trying to setup k3s HA on 4 raspberry pis. After several hours I eventually gave up and decided to try microK8s and it worked instantly. 🤷♂️