Your restart policy is deleting the crime scene

Automatic restart policies are useful right up until they become a document shredder.

The service crashes. The supervisor restarts it. It crashes again. The supervisor restarts it again. By the time the pager fires, the original failure is buried under thirty nearly identical startup messages, the process ID has changed repeatedly, temporary files are gone, and the one useful core dump has been rotated out because the machine is trying very hard to be helpful.

Availability matters. So does evidence.

Recovery and diagnosis are different jobs

A restart policy answers one question:

Can the service return to operation without a human?

It does not answer:

Why did the service fail?

Those goals overlap, but they are not identical. A supervisor optimized only for recovery can destroy the context needed for diagnosis.

For a systemd unit, a common policy looks like this:

[Service]
Restart=on-failure
RestartSec=2s

That may be correct for a transient network error. It may be disastrous for a deterministic crash that occurs two seconds after startup.

The process can churn continuously while producing no useful new information.

Put a ceiling on enthusiasm

Systemd has start-rate limiting for exactly this reason:

[Unit]
StartLimitIntervalSec=5min
StartLimitBurst=5

[Service]
Restart=on-failure
RestartSec=10s

This permits recovery attempts but stops an endless loop after repeated failures in the configured interval.

The exact values depend on the service. A stateless worker may tolerate several quick retries. A database, message broker, or process that performs migrations during startup deserves more restraint.

Do not copy limits from an example and call it engineering. Choose values based on how long startup takes, how expensive failure is, and how much evidence each attempt overwrites.

Keep the first failure visible

The first crash often contains the cleanest signal. Later crashes may be consequences of damaged state, half-completed initialization, locked resources, or recovery logic.

Start with the unit’s journal around the first failure:

journalctl -u example.service \
  --since '2026-08-05 08:00:00' \
  --until '2026-08-05 08:10:00' \
  --output=short-precise

Use precise timestamps because “it failed around eight” is not a timeline.

Inspect the service’s recent exit state:

systemctl show example.service \
  -p Result \
  -p ExecMainCode \
  -p ExecMainStatus \
  -p NRestarts

NRestarts is especially useful when the dashboard says one incident but the supervisor has actually attempted recovery dozens of times.

Preserve core dumps before retention wins

If the process dumped core and the host uses systemd-coredump:

coredumpctl list example

Inspect metadata:

coredumpctl info <PID-or-COREDUMP-ID>

Export a core before cleanup or size limits remove it:

coredumpctl dump <PID-or-COREDUMP-ID> \
  --output=/var/tmp/example-first-failure.core

Then record the executable build, package version, container image digest, configuration checksum, and relevant library versions. A core without the matching binary and symbols can become an impressively large mystery file.

Core dumps may contain credentials, user data, encryption material, and in-memory secrets. Restrict access and follow the same handling rules you would apply to sensitive production data.

Capture state before another restart

Some failures leave useful state outside the process:

• temporary files
• Unix sockets
• lock files
• queue depth
• open file counts
• kernel messages
• cgroup memory events
• container exit metadata
• recent deployment identifiers

Before manually restarting, collect the pieces that will change:

stamp=$(date -u +%Y%m%dT%H%M%SZ)
out=/var/tmp/example-incident-$stamp
mkdir -m 0700 "$out"

systemctl status example.service --no-pager >"$out/status.txt" 2>&1
systemctl show example.service >"$out/show.txt" 2>&1
journalctl -u example.service --since '-15 min' >"$out/journal.txt" 2>&1
dmesg --ctime | tail -n 300 >"$out/dmesg-tail.txt" 2>&1

That is not a complete forensic collection. It is a practical minimum that survives the next recovery attempt.

Use commands appropriate to the host and privilege model. Do not dump secrets into a world-readable directory because an incident made everyone hurried.

Make the supervisor escalate instead of spin

A mature restart policy has an escalation path.

After repeated failures, the system should do something different:

• stop restarting
• emit a distinct alert
• preserve logs and cores
• mark the instance unhealthy
• remove it from service
• require an operator or controlled remediation workflow

The important part is the state change. Repeating the same failed action faster is not resilience.

For some services, OnFailure= can trigger a separate unit that collects diagnostics or sends a high-signal alert. Keep that failure handler small. An elaborate incident script with twelve network dependencies is another service waiting to fail during the exact conditions it was meant to observe.

Do not disable restarts everywhere

The lesson is not that automatic recovery is bad.

A process killed by a transient resource condition may recover cleanly. A worker can reconnect after a temporary dependency outage. A daemon may fail once during a package transition and run normally afterward.

The lesson is that recovery needs boundaries:

1. Retry a limited number of times.
2. Slow down repeated attempts.
3. Preserve the first useful evidence.
4. Alert when the failure becomes persistent.
5. Stop pretending a restart loop is healthy service.

A green supervisor state after the thirty-first restart is not stability. It is a machine repeatedly sweeping broken glass under the same rug.

Keep the restart policy. Just make sure it leaves a crime scene behind.