A shop I was helping had a billing export in root’s crontab: 30 2 * * *, run at 02:30, push a CSV to the payment processor’s SFTP drop. It had worked for four years. The server’s timezone was America/New_York, because somebody installed it by clicking through the defaults and nobody ever looked again.
On the first Sunday of November the wall clock went from 01:59 back to 01:00. For one night, 01:30 existed twice. Depending on the cron implementation and its version, a job pinned to a time inside that repeated hour can run once or twice, and the customer’s setup ran it twice. Two exports, two uploads, and a processor that cheerfully ingested both. The refund queue the following Monday was not cheerful.
What cron actually promises
Less than you think. The classic Vixie-derived crons special-case fixed-time jobs during small clock jumps, so that a job at a specific hour and minute is meant to run once, not twice, and not zero times. Jobs with a wildcard in the hour or minute field are treated as “every period” and simply follow the wall clock. Other implementations, including some minimalist ones and some distro forks, do not do this, or do it with different edge cases.
I’m not going to quote you a version number from memory. Read the cron(8) page that shipped with your system, and test on a scratch box by moving the clock. The point is that “cron handles DST” is a claim about one implementation, and the box in the story was running a different one than the author of the job assumed.
The spring side is meaner. On the night clocks jump from 02:00 to 03:00, 02:30 never happens at all. A fixed-time job in that gap is either run late by an implementation that compensates, or silently skipped by one that doesn’t. A skipped nightly job produces no error. It produces a missing file, discovered when somebody asks where it is. See the alert that fired for eight months for how long “nobody noticed” can last.
The same family of bug as the machine that forgot the year
Time on a computer is a set of agreements between the kernel, libc, the zone database and whatever reads the clock. Break one and you get a very confident wrong answer. I wrote up the machine that forgot what year it was earlier, and this is its cousin: nothing was broken, the assumption was just local.
Local time is a presentation layer. It isn’t monotonic, it isn’t unique, and twice a year it isn’t even contiguous. A scheduler keyed on it inherits all three properties.
How I’d have found it quickly
After the fact, the evidence was in the logs. Two lines, same job, an hour apart by the wall clock but timestamped identically to the minute:
grep CRON /var/log/cron | grep billing-export
date +%Z
zdump -v America/New_York | grep 2026
zdump -v prints every transition for the zone in the database, which is how you confirm the exact moment the repeated hour happens instead of arguing about it. Log timestamps without a zone offset are useless on this night, so check whether yours carry one.
The fix, in order of how much I trust it
- Run servers in UTC.
timedatectlon systemd boxes, the/etc/localtimelink andtzsetupor an/etc/localtimecopy on FreeBSD. UTC has no gaps and no repeats. If a human wants the job at “2:30 local”, they can do the arithmetic once and write it in a comment. This is the fix I would apply on every server I own, and it is the one the shop took. - Make the job idempotent. The export should have written a dated marker, or named the file by the business date, and refused to upload a second copy. Even in a UTC world, a retry or a double-fire will eventually happen, and “run twice” should be boring. The same principle applies to the cron job that ran twice for a completely different reason.
- Avoid the transition window if you can’t do either. Don’t schedule fixed-time jobs between 01:00 and 03:00 local. Cheap, effective, and relies on you remembering why.
Some cron implementations support a per-crontab timezone variable. That’s useful for a job that truly must follow a business clock, but it has the same sharp edges at the transition, so it moves the problem rather than removing it. Check your own crontab(5).
What I took from it
Every scheduled job has an implicit contract: it runs once per period. DST is one of the few events that breaks the contract on a perfectly healthy machine, on a date you can look up years in advance. There’s no excuse for being surprised, and yet it surprises a shop somewhere every spring and fall.
Put the clocks in UTC, make the batch idempotent, and write the file name with the date in it. If you want more stories where a healthy-looking system quietly did the wrong thing, the reliability hub has the pile.
