Your shell pipeline succeeded because the last command was polite

Here is a shell script that can destroy your evening while returning zero:

pg_dump production | gzip > backup.sql.gz

If pg_dump dies because the database connection drops, gzip may still consume the partial stream, close the output file, and exit successfully. The pipeline’s status is normally the status of the last command. Your cron job records success. Your monitoring stays green. Your backup is a compressed account of how little data you still have.

The shell did exactly what you asked. That is the problem.

The default rule is older than your incident process

Traditional shells treat a pipeline as one compound command and report the exit status of its final component:

false | true
echo "$?"
# 0

This is useful when the consumer’s result is what matters. It is dangerous when every stage is part of the contract.

A surprisingly large amount of production shell code assumes that set -e fixes this. It does not. Without pipeline-aware behavior, the pipeline succeeded because true succeeded. errexit has nothing to object to.

Bash, ksh, and zsh: turn on pipefail

In shells that support it:

set -o pipefail
pg_dump production | gzip > backup.sql.gz

With pipefail, the pipeline returns failure when any component fails. In Bash, the status is the value of the rightmost command that exited nonzero, or zero if all commands succeeded.

For scripts explicitly written for Bash, I usually start with:

#!/usr/bin/env bash
set -Eeuo pipefail

Then I immediately remind whoever is reading it that this is not a magic correctness incantation. -e has context-dependent exceptions. -u can punish legitimate optional variables. pipefail tells you that something failed, not which thing or whether cleanup happened.

Flags are guardrails. They are not design.

When you need the individual statuses

Bash exposes an array named PIPESTATUS immediately after a foreground pipeline:

producer | transform | consumer
status=("${PIPESTATUS[@]}")

printf 'producer=%d transform=%d consumer=%d\n' \
  "${status[0]}" "${status[1]}" "${status[2]}"

Copy it immediately. Running another command—even echo—replaces the values because that command becomes the most recent pipeline.

This matters when failure meanings differ. A producer returning 1 may mean “no matches,” while a consumer returning 1 means “disk full.” Collapsing both into a generic pipeline failure is better than reporting success, but it is still not enough for useful operations.

POSIX sh does not promise pipefail

Do not put set -o pipefail into a script with #!/bin/sh and assume the machine agrees with you. pipefail is not required by POSIX. Some /bin/sh implementations support it; others do not. FreeBSD’s sh supports pipefail, but portable scripts cannot rely on every target doing so.

For truly portable code, avoid the pipeline when you must preserve each status. Use temporary files, named pipes with explicit process management, or restructure the work so commands run separately:

tmp=${TMPDIR:-/tmp}/dump.$$
trap 'rm -f "$tmp"' EXIT HUP INT TERM

if ! pg_dump production > "$tmp"; then
    echo 'database dump failed' >&2
    exit 1
fi

if ! gzip -c "$tmp" > backup.sql.gz; then
    echo 'compression failed' >&2
    exit 1
fi

Yes, it writes an intermediate file. Yes, that may be unacceptable for a giant database. Engineering remains the irritating process of choosing which constraint you are actually solving.

For streaming plus portability, supervise the processes explicitly in a language with sane process APIs. Shell is glue. When your glue starts needing a scheduler, a state machine, and forensic telemetry, congratulations: you have invented a worse programming language.

Validate the artifact, not merely the command

A zero exit code proves only that the programs followed their implemented paths to success. It does not prove the resulting backup is complete, current, restorable, or even pointed at the database you meant.

For backups, add checks that test the thing you care about:

• reject empty or implausibly small output
• verify compression integrity with gzip -t
• record source identity and timestamps
• periodically restore into an isolated environment
• alert on age, not only job failure

A backup job that exits zero is an opinion. A restore test is evidence.

The useful habit

Whenever you see a pipeline in automation, ask one question:

If an earlier command fails and the final command exits zero, will we know?

If the answer is no, the script is not concise. It is merely withholding information.

Turn on pipefail where the shell contract allows it. Capture individual statuses where they matter. Restructure portable scripts when they do not. Then verify the artifact, because the shell’s idea of success has never been the same thing as yours.