What exactly do you wish to get input on? Reliability, cost effectiveness, availability? You also mention maintaining production hardened hardware?
Let's start somewhere with one of these.
Availability of services in cluster is reliant on many things:
- Load Balancing
- Rolling updates
- Replication
- Scaling
- etc
Reliability is ensuring efficient monitoring, good recovery and fallback mechanisms, self healing, etc. Realistically you need to make sure you have n amount of replicas of a service in multiple physical locations, all backed up.
Cost effectiveness involves optimizing resource utilization, scaling efficiently, and managing infrastructure in a way that maximizes value for the resources consumed.
My suggestion:
- Automate creation of the cluster. If you want to learn industry standards, you can take a look at terraform or ansible and one of the public cloud providers. K8s was designed to run on a cloud provider ( even tho it can also be ran locally ofc ).
- Deploy a CSI
- Ingress
- Certificates and ssl
- Deploy a monitoring solution and make sure that it survives you unplugging one of the servers from the plug.
- Repeat step 5 for other apps and stumble through the unique challenges that you'll face with scaling most of them
Side Note: take a look at k3s