Your cron job is running twice and that's why the numbers are wrong

The report job runs every five minutes and takes four. Everything is fine for eight months. Then the database gets bigger, one run takes six minutes, and now two copies are running at once — reading the same rows, writing the same output file, incrementing the same counters.

Nothing crashes. The numbers just quietly stop being right, and by the time anyone notices, the wrong numbers are three weeks into a dashboard somebody makes decisions from.

The one-liner

*/5 * * * * /usr/bin/flock -n /var/lock/report.lock /usr/local/bin/report.sh

flock takes an exclusive lock on the file and runs the command. -n means “don’t wait” — if the lock is held, exit immediately (status 1) and don’t run. That’s the behavior you want for a periodic job: the previous run is still going, so skip this one, it’ll come around again in five minutes.

The lock is released when flock’s child exits, including if it’s killed, including if the box loses power. That’s the entire reason to use it instead of what you were about to write.

The pidfile you were about to write is broken

Everyone reinvents this:

if [ -f /var/run/job.pid ]; then
    echo "already running"; exit 1
fi
echo $$ > /var/run/job.pid
trap 'rm -f /var/run/job.pid' EXIT

Three bugs, and they’re all the kind that surface at the worst time:

It’s a race. Two invocations can both pass the -f test before either writes. Test-and-set is not atomic in shell, and no amount of rearranging makes it so. flock(2) is a single kernel operation; there is no window.

It leaks on SIGKILL. trap ... EXIT doesn’t run on SIGKILL, and it doesn’t run if the machine reboots. Now the pidfile is stale, the job never runs again, and nobody notices until somebody asks why last month’s report is missing. This is the most common way this pattern fails.

PIDs get reused. Check “is the PID in the file still alive” and you’ll eventually find a completely unrelated process wearing that number, and decide your job is still running when it finished on Tuesday.

flock has none of these. The lock is a property of the open file description held by the kernel, so when the process dies — however it dies — the lock goes with it. Stale lock files are impossible by construction. The lock file itself persisting is fine and expected; it’s an inode to lock, not a flag.

Variations worth knowing

# Wait up to 60s for the lock, then give up (for jobs that must run, eventually)
flock -w 60 /var/lock/job.lock /usr/local/bin/job.sh

# Lock inside the script instead of in crontab — self-contained, survives being
# invoked by hand
#!/bin/bash
exec 9>/var/lock/job.lock
flock -n 9 || { echo "already running"; exit 0; }
# ... work ...

That exec 9> idiom is the good one for scripts. It opens fd 9 on the lock file for the life of the script, locks it, and never has to clean up. Note the exit 0 — a skipped run is usually not an error, and returning 1 to cron means an email every five minutes forever, which trains everyone to filter cron mail, which is how you stop noticing the real failures.

Lock a directory when the resource is a directory, not a file:

flock -n /var/lib/myapp -c 'rsync -a /var/lib/myapp/ /backup/'

Different jobs, same resource — use the same lock file across all of them. The lock is on the contested resource, not on the job. Three different scripts that all rebuild the same index should share one lock, and this is the part people get wrong: they give each script its own lock file, which locks nothing meaningful.

The limits, stated plainly

flock is advisory and local to one machine. It works because every participant agrees to ask for the lock. A process that doesn’t call flock walks straight past it.

And on NFS: flock on Linux over NFSv4 maps to POSIX locks and generally works, but “generally” is carrying weight there — client caching, server restarts and lock recovery all have edge cases. If you need mutual exclusion across hosts, use something built for it: a lock row in the database you’re already using, a lease in etcd/Consul, or a queue that hands out work. Do not use a lock file on a shared mount and hope. That is a distributed systems problem wearing a filesystem costume, and it will find you during a network partition.

While you’re in there

Two other things about that crontab line, since you’re editing it:

*/5 * * * * /usr/bin/flock -n /var/lock/report.lock /usr/local/bin/report.sh 2>&1 | logger -t report

Pipe to logger instead of letting cron mail you. Now the output is in syslog with a tag, timestamped, rotated, and greppable alongside everything else — instead of in a mail spool nobody has opened since 2019.

And set PATH at the top of the crontab. Cron’s environment is nearly empty, which is why the script works when you run it and fails at 2am. That, and flock, cover the overwhelming majority of “the cron job is behaving strangely” tickets.