I'm completely OTel with Prometheus, Jaeger, and Loki now. No reason to fart around with dated collection/analysis tools with data silos -- OTel is good enough for work, it's good enough for the lab. I say this as someone who used Nagios and MRTG for many years, and have contributed to Nagios Core.
I'm pretty comfortable working with Ansible and Chef, though everything except the network equipment in my lab runs on k8s and Talos now. I've managed fairly complex Prometheus and Nagios configs with Chef and Ansible in the past -- not too tricky once you have a config management tool on your belt IMO.
Yup. Just my $0.02 -- I wouldn't say there's significant benefit over running a mainstream Linux distro and dropping something like RKE2, k3s, Kind, etc on the hardware. If/when I rebuild this cluster, I'll likely switch back to my existing Ansible roles for RKE2 on Ubuntu. I think the intersection of "people who need a very thin distro for just k8s" and "people running baremetal self-managed k8s" is pretty narrow. All the major cloud vendors already produce a thin distro of their own for "just k8s". Choosing Talos likely adds up fast when you're talking hundreds of clusters and thousands of nodes in colos or on-prem.
I have an ARM based cluster I use for network/CNI labbing which runs k3s on Armbian.