49
How do you handle cron jobs that silently stop working? (lemmy.world)
submitted 3 months ago by to c/programming@programming.dev
15 comments fedilink hide all child comments
 

Had a backup script running via cron for months. Worked fine until it didn't — turns out the disk filled up three weeks ago and the job started failing silently. Nobody noticed until we actually needed a restore.

The obvious answer is "check your logs" but let's be honest, nobody's reading cron logs daily for 15 different scheduled tasks across 4 servers.

What's your setup for making sure crons are actually completing? Do you just grep logs periodically, or do you have something more structured? Curious how others handle this without turning it into a whole project.

you are viewing a single comment's thread
view the rest of the comments
[–] 5 points 3 months ago (3 children)

Monitor your logs with a network monitoring tool like Nagios, Icinga, etc. You can set filters for alerts you are genuinely interested in, and email the alerts to your ticketing system or sysads.

  • permalink
  • fedilink
  • source
  • hideshow 3 child comments
    •
  • [–] 3 points 3 months ago (2 children)

    Graylog is excellent for catching all the logs from different system and filtering them according to your tastes into email alerts.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 1 point 3 months ago (1 child)

    Not used graylog, but if that's what it's for, it should be a lot better than manually checking logs.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 3 months ago

    Graylog is excellent for rummaging about in log files. Client side you have to install rsyslog and one universal config file to forward all the logs to a central server. Gralog server-side has a GUI where you can set up email alerts by basically setting up greps on stuff from every log file on your whole network. Some advanced functions of graylog are behind a paywall but I have never needed them.

  • permalink
  • fedilink
  • source
  • parent