c/programming · by indidev@lemmy.world · 9d

cron PATH bit me again today

Deployed a backup script on a new server, tested it manually — worked fine. Set up crontab, came back next morning, no backup. Turns out /usr/local/bin wasn't in cron's PATH so pg_dump just silently didn't exist.

Switched to absolute path and it worked immediately. Fifteen minutes of debugging for a one-character fix. I keep making this mistake every time I set up a new box. At this point I should just have a checklist taped to my monitor.

32 points · 11 comments · view on lemmy.world

11 Comments

Skullgrid@lemmy.world · 14 pts · 9d

hey, at least you caught it within a day.

EDIT : Next time make the cron job run when you're finished deploying it.

bjoern_tantau@swg-empire.de · 14 pts · 9d (1 reply)

That's one reason why I like using systemd timers instead of cronjobs. You can easily test it by running the unit instead of setting the job to "in a minute".

Downside is that you don't easily get notification mails for any output.

dan@upvote.au · 2 pts · 9d

I use healthchecks + runitor for notifications. Would recommend. I posted another comment about them.

runwisp_com@lemmy.world · 7 pts · 9d

cron isn't using your login shell. that's the trap.

but put PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin at the top of the crontab, then test the exact command once with env -i HOME="$HOME" PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin /bin/sh -c '...', and you catch most of the "worked in my terminal" stuff before it turns into tomorrow's missing backup. MAILTO or a heartbeat catches the next failure.

87Six@lemmy.zip · 4 pts · 7d (1 reply)

This bit me so many times I have

# USE ABSOLUTE PATHS, ASSUME ALL COMMANDS ARE UNDEFINED, SPECIFY ALL VERSIONS, REDIRECT OUTPUT

at the top of my local crontab so I remember from when I take something from it to prod, lol.

Edit: formatting whoopsie

Kissaki@programming.dev · 2 pts · 7d

When will this be part of the packaged default? :)

dan@upvote.au · 3 pts · 9d

I'd recommend self-hosting healthchecks and using runitor for your cronjobs. Runitor pings healthchecks when the cronjob starts, then pings it again on completion with the status (success or fail) along with the stdout and stderr.

Healthchecks can be configured to expect a ping periodically (once per day, once per hour, whatever) and alert you if it doesn't receive one.

For backups, Borgmatic has a healthchecks integration.

marcos@lemmy.world · 3 pts · 9d (2 replies)

Looks like you need to collect your crontab logs somewhere you can read.

folekaule@lemmy.world · 5 pts · 9d

Or maybe even set up a valid MAILTO in the crontab so that failures are emailed to them.

My "pro" tips:

set up an email alias from root to an email you actually read
always use absolute paths
anything complex, put it in a shell script

pelya@lemmy.world · 3 pts · 9d

You can append

2>&1 | logger -t my-cronjob

to any command, and it will write logs to system journal which you can view withjournalctl

grrgyle@slrpnk.net · 1 pts · 8d

You know I'd totally forgot about this but you're reminding me of my own insane debugging experiences over a decade who