Running services, network usage, memory usage, bandwidth, disk I/O, successful logins, whether the thing is even alive, etc...
So far my only method has been "hope and pray".

you are viewing a single comment's thread
view the rest of the comments
[–] 3 points 1 week ago

I use Signoz and OTel collectors to bring in everything from Kubernetes (and also SNMP-to-OTel for the network and NAS stats.)

That covers dashboards and basic alerting. Then on top of that I have an n8n workflow that runs every morning - it pulls all the main stats out of Signoz, as well as issues and their status from my infrastructure repo in Gitlab, and feeds it all into a local LLM (Qwen-3.8-Flash-Next at the moment), which emails me a report on overall cluster health. It also automatically opens a ticket in Gitlab for anything that might need fixing.

It works really nicely, gotta say.

  • source