A junior wrote a nightly database dump, tested it by hand, watched a dated file appear, and pasted the line into the crontab. Then he went home feeling good about himself. Reasonable. I’ve done the same thing and felt the same warmth.
Five weeks later somebody needed last Tuesday’s dump. The backup directory was empty. Not stale, not truncated. Empty, as if the job had never been scheduled, except it plainly had been, because crontab -l showed it sitting there, smug.
The line
15 2 * * * pg_dump appdb | gzip > /backup/appdb-$(date +%F).sql.gz
Typed into a shell, that works every time. date +%F expands to 2026-11-09, the pipe does its thing, you get a file. Nothing about it looks wrong, which is why three people read it in review and approved it.
Cron is not a shell. Cron is a program that reads a table, and the table has its own quoting rules that predate most of the people reading this. One of them: an unescaped % in the command field means newline. Everything before the first % is the command. Everything after it is fed to that command as standard input, with each further % turning into another newline.
So cron did not run the line above. It ran this:
pg_dump appdb | gzip > /backup/appdb-$(date +
with the string F).sql.gz piped in on stdin. That is an unterminated command substitution. sh looked at it, said something about an unexpected end of file, and exited non-zero. No dump started. No file was created. The shell never got far enough to open the redirect.
Why nobody heard about it
Cron’s reporting channel is mail. Anything a job prints to stdout or stderr gets mailed to the owner of the crontab, and the syntax error was printed to stderr. Every night, for five weeks, a polite little message landed in root’s local mailbox.
Root’s mail on that host was not forwarded anywhere. It was a spool file in /var/mail, and by the time anyone opened it, it was a few hundred identical messages about a syntax error. Reading it took four seconds. Nobody had read it in the ordinary course of life because nobody reads root’s mail, which is a sentence that explains roughly a third of all the outages I’ve been paid to clean up.
The job also had no exit-status monitoring and no check on the output, so there was nothing positive to notice the absence of. A backup that fails loudly is an annoyance. A backup that never starts and is never missed is a policy decision nobody made. If this sounds familiar, it’s the same family as the backup that restored perfectly and the daemon that wrote the files while the backup was empty. The tool did exactly what it was told. Someone just never confirmed it was told the right thing.
Reproducing it without waiting for 02:15
You don’t have to wait for the schedule to find out whether a crontab line is sane. Do what cron does to the line and look at the result:
printf '%s\n' 'echo today is $(date +%F)' | sed 's/%.*//'
That’s a crude demonstration, but it shows the point: strip at the first % and see what’s left. If what’s left isn’t a complete command, cron will run something other than what you meant.
A better test is to temporarily schedule the line for the next minute, with output going to a file you will actually look at:
* * * * * pg_dump appdb | gzip > /backup/appdb-$(date +%F).sql.gz 2>/tmp/cron-test.err
Wait sixty seconds, cat /tmp/cron-test.err, then take the test line out. Crude, and it works. The part people skip is taking the test line out, which is its own small war story.
The fixes, in order of how much I like them
Escape the percent signs. In a crontab, \% is a literal percent:
15 2 * * * pg_dump appdb | gzip > /backup/appdb-$(date +\%F).sql.gz
That is the minimum. It works with Vixie-style cron on the BSDs and the common Linux crons. Check your own cron’s man page (man 5 crontab) rather than trusting me or a forum post, because the exact rules for what gets escaped and what reaches stdin are in there.
Better: don’t put logic in the crontab at all. Put the command in a script, give it a real shebang, set -eu, and make the crontab line a single path with no punctuation in it:
15 2 * * * /usr/local/sbin/appdb-dump
Now you can run the exact thing cron runs, by hand, with the same arguments, and % is just a character in a file. The script is also where the sanity check belongs: after the dump, refuse to call it a success unless the file exists and is bigger than some floor you chose on purpose. If the dump can silently produce nothing, the script is lying to you.
Make failure reach a human. Set MAILTO in the crontab to an address that someone actually reads, or have the script ping a dead man’s switch on success so that silence is the alarm. Silence from a job is not a signal unless something is waiting for the signal. For overlapping runs, which is the opposite problem, there’s the flock tip, and the cron job that ran twice is the same class of surprise from the other direction.
What I’d tell the junior
Nothing cruel. He did what the whole industry teaches: test it, then schedule it. The test was just against the wrong interpreter. A line that works in your shell has proven it works in your shell.
The rule I use is dull and it has never failed me: the crontab holds a schedule and a path, nothing else. Anything with a $, a %, a quote, or a pipe in it lives in a file that I can run with sh -x and read like an adult. More of this kind of thing lives in the reliability hub, mostly because production keeps being cheerful about not telling us things.
