timeout(1) needs -k, and exit 124 is not a failure you can ignore

Every hung production job I have ever been paged for had one thing in common: nobody gave it a deadline. A curl with no --max-time, an ssh to a box that was already dead, a backup script waiting on an NFS mount that went away. The process was not slow. It was waiting forever, politely, and it was holding a lock the whole time.

timeout(1) fixes this in one word. It is in GNU coreutils and in the FreeBSD base system, and almost everyone uses it wrong.

The basic form

timeout 30 pg_dump mydb > /backup/mydb.sql

After 30 seconds, timeout sends the command SIGTERM. Durations take s, m, h or d suffixes, seconds by default, and 0 means no limit. -s picks a different signal, so timeout -s HUP 1h ./reload-loop works if that is what the child understands.

Why you need -k

SIGTERM is a request. A process that traps it, ignores it, or is stuck in uninterruptible sleep on a dead mount does not have to honor it. Plain timeout 30 cmd then sits there with you believing a deadline exists.

timeout -k 10 30 ./nightly-sync.sh

That reads as: after 30 seconds send SIGTERM, and if the command is still alive 10 seconds later, send SIGKILL. SIGKILL cannot be caught. The only thing it cannot fix is a process in D state waiting on I/O that the kernel will not abandon; that one needs the mount fixed, and I wrote about finding it in reading a wedged process without a dashboard.

The exit codes you must check

This is the part people skip, and it is why a “timed out” job later looks like a successful one.

Status	Meaning (GNU coreutils)
124	Command timed out
125	timeout itself failed
126	Command was found but could not be invoked
127	Command not found
137	SIGKILL was sent via -k (128 + 9)
other	The command’s own exit status

FreeBSD’s timeout uses 124 as well, so the same script runs on both. Check it explicitly:

timeout -k 10 30 ./nightly-sync.sh
rc=$?
case $rc in
  124|137) echo "nightly-sync exceeded its deadline" >&2; exit 1 ;;
  0)       ;;
  *)       echo "nightly-sync failed: $rc" >&2; exit "$rc" ;;
esac

If you use --preserve-status, timeout exits with the child’s own status after signaling it instead of 124, which is useful when a wrapper cares about the real code but means you can no longer tell a timeout from an ordinary failure by the exit value alone. Pick one on purpose.

Inside a pipeline the 124 vanishes unless you handle pipeline exit status with pipefail or PIPESTATUS. timeout 5 slow | head returns head’s status and your deadline was a decoration.

—foreground and the TTY trap

By default timeout runs the command in its own background process group, so that on expiry it can signal the command and everything it spawned. That is exactly what you want in cron. It is exactly wrong when you run something interactive:

timeout 300 ssh somebox     # hangs, or the prompt never reads your keystrokes

The child is in a background group and gets stopped when it tries to read the terminal. --foreground skips the group dance so the command keeps the TTY and gets Ctrl-C normally. The documented catch is that children of the command are no longer timed out, only the command itself. For a script that forks workers, that gap matters, so use it for interactive tools and not for process trees.

Pair it with a lock

A deadline and a lock belong together. A job that cannot overlap and cannot run forever is a job that fails loudly instead of piling up:

flock -n /run/nightly.lock timeout -k 10 3600 ./nightly-sync.sh

The lock stops the second run from starting, and the timeout stops the first from living until the heat death of the host. The flock post covers why -n is the part that matters. If your script needs to clean up when timeout terminates it, install a handler as described in trap and signal handling; SIGKILL skips traps entirely, which is another reason to keep the -k grace period long enough for cleanup.

The cons

• A timeout kills the work, it does not roll it back. A half-written dump file from a killed pg_dump is still on disk. Write to a temp name and mv on success.
• It guards wall-clock time, not CPU or I/O. A process that is busy but pointless still gets the full allowance.
• Long-running legitimate jobs need a number someone chose on purpose. Set it from the observed p99 plus margin, not from a guess, or the kill will land at the worst moment of a slow month-end.
• It only wraps a command. Timeouts inside the program (connect timeouts, read timeouts) still belong in the program; timeout is the backstop, not the design.

The old rule stands: anything that talks to a network or a disk without a deadline is a bet that the other side never fails. The other side always fails eventually, usually on a Friday.