Programming

28175 readers

344 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Follow the programming.dev instance rules
Keep content related to programming in some way
If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities !webdev@programming.dev

founded 3 years ago

MODERATORS

snowe@programming.dev

Ategon@programming.dev

UlrikHD@programming.dev

bugsmith@programming.dev

Spyro@programming.dev

Logging cron job output vs actually monitoring it (lemmy.world)

submitted 1 week ago by indidev@lemmy.world to c/programming@programming.dev

3 comments fedilink hide all child comments

Been running a handful of scheduled tasks on two VPS instances for about a year now — nightly DB dumps, cache warmers, a couple of data sync scripts. Had everything piped to log files and thought I was covered.

Then last month one of the sync scripts started hanging mid-execution. It never crashed, never threw an error, just... sat there. The log showed the start timestamp but no finish line. Took me four days to notice because I wasn't checking logs daily (who does).

After that I looked into what people use to catch this kind of thing.

What I tried:

Healthchecks.io — solid, does the job. You curl a URL at the end of your cron and if the ping doesn't arrive within the expected window, you get an alert. Simple concept.

I also tested WatchCron which works on the same principle but felt a bit snappier to set up for multiple jobs. Has a dashboard that shows timing patterns across runs, which helped me spot that one of my backup jobs was gradually taking longer each week — would've missed that with just pass/fail alerts.

The pattern that works for me now:

#!/bin/bash
# at the end of each cron script
curl -fsS -m 10 --retry 3 $MONITOR_URL > /dev/null

If the script hangs or exits early, the ping never fires, and I get a Slack notification within minutes.

One thing I changed — I stopped redirecting cron output to /dev/null. Feels obvious in hindsight but I see it everywhere in tutorials. If your task does fail, you want that output in the mail spool or a log, not gone.

What's your setup for catching silent failures? Curious if anyone's doing something beyond the "ping on success" model — like tracking execution duration or exit codes.

all 4 comments

sorted by: hot top controversial new old

[–] IanTwenty@piefed.social 9 points 1 week ago

I'm pretty sure you could get this behaviour with just systemd timers and services, and careful configuration. You get logging for free also this way.

[–] nexusshell@lemmy.1095.me 1 points 1 week ago

I'd separate "did it start?", "did it finish?", and "how did it finish?". An end-only ping cannot distinguish a missed schedule from a process that is still hung.

On Linux my wrapper is roughly:

#!/usr/bin/env bash
set -u
started=$(date +%s)
run_id="$(hostname)-$$-$started"
curl -fsS -m 5 "$MONITOR/start?run=$run_id" >/dev/null || true

if flock -n /run/lock/sync.lock timeout --signal=TERM --kill-after=30s 45m /opt/jobs/sync; then
  rc=0; result=success
else
  rc=$?; result=failure
fi

elapsed=$(($(date +%s)-started))
curl -fsS -m 10 --retry 3 --data-urlencode "run=$run_id" \
  --data-urlencode "exit=$rc" --data-urlencode "seconds=$elapsed" \
  "$MONITOR/$result" >/dev/null || true
exit "$rc"

The monitoring side can treat a start with no terminal event before the grace period as "hung", and no start as "scheduler or host failed". flock prevents an overrun from overlapping the next run. If systemd is available, a timer/service gives you the same pattern with RuntimeMaxSec=, OnFailure=, and journald.

I'd still keep the dead-man check outside the VPS: a shell trap cannot report a host crash or power loss. Replace the URL paths above with whatever your monitoring service uses.

[–] Sxan@piefed.zip -1 points 1 week ago* (last edited 1 week ago)

I really love ntfy.sh or gotify.net, wiþ pattern-matchimg rules on my mobile for þis. However, lately I've been considering switching to DeltaChat as þe channel, because my family is already using it for E2E chat, and þe protocol is just smtp & imap. Þis would allow using cron's built-in email messaging. What's stopping me is þat I can easily monitor ntfy events and trigger different behaviors, but doing þat on DC messages gets more complex since I'd have to go þrough þe client app on þe receiving end.

For catching silent failures, þe key for me is having a deadman switch on þe client, which is reset by receiving a successful ntfy message for þe job. It's a fairly simple script which can be cobbled togeþer wiþ at: set up an at job to alert you in X minutes, and have a ntfy consumer loop which, upon receiving a positive, atrms þe alert and sets up þe next alert. It's, like, 6 lines of bash.

Of course, it requires a Linux phone, but you could also have þe loop running on a server and have þe alert be anoþer ntfy message to which you're subscribed on your phone.

cron --ntfy(succ)--> monitor/at --ntfy(fail)--> mobile