IMO, you really don't want to run databases over NFS/SMB. I've just seen too many corruption and locking issues over the years to trust it.
For home lab use, do you really need anything as big enough for a database that wouldn't fit on a Pi anyway?
For example, I run Prometheus on my Pi, no problem.
Check out the smokeping_prober. It generates high resolution histogram data that you can use to compare different paths.
I say application monitoring is the key thing mainly because what we actually want to know is, is the system as a whole functioning correctly? Without positive evidence that the application is succeeding we can't tell for sure.
So true.
The problem was at the time when Nagios was introduced the only metrics system we had was SNMP. SNMP was kinda ok to get into with the introduction of Cacti. But due to the complication of ASN.1, the split between the actual data and the MIBs, and the lack of good documentation on how to implement it properly, it was not a good solution to extend. Adding application metrics in SNMP? Good luck with that.
Thankfully then came statsd, and then Prometheus, to make it much easier to implement application monitoring, as well as integrate system and network monitoring with application monitoring.
rsyslog/syslog-ng are simple enough.
But logging and monitoring/alerting are different things. Logging is not great for alerts. You want an actual monitoring system for that.
Prometheus is not that complicated, has a huge community of integrations, and is extremely flexible for monitoring everything.
I use a Pro card meant for cameras. Also, Prometheus disk write load is tiny.