Topic · 28 articles
Reliability & Observability
Most of the outages in here were caused by something working exactly as designed. The retry logic retried. The health check passed. The alert fired, every night, for eight months. The failure was always in what someone assumed the green light meant.
This hub is mostly Tales, because that's where the lessons live. Start with the four that generalise furthest, then the Tips that would have caught them: supervision that keeps the evidence, cron that can't overlap, and pipelines that can't lie about their exit status.
Start here
Read these first, in this order.
- 01
Tale · 2026-08-04
The health check was green because it was checking the wrong damn thing
A service can answer HTTP 200 while its queue is wedged, its database writes are failing, and every useful request is dying. RodHat on health checks that prove process existence instead of service capability.
- 02
Tale · 2026-08-01
We wrote the retry logic to survive a blip. It turned a blip into four hours.
One backend got slow for ninety seconds. Every client retried, in lockstep, three times, with no jitter. The retries were larger than the original traffic, the backend never recovered, and every fix we tried made it worse until we did the thing nobody wanted to do.
- 03
Tale · 2026-08-10
The change was correct. Deploying it to every region simultaneously was not.
A one-line config change, reviewed, tested, and correct in every environment we tried it in. It went to all six regions in the same rollout because config isn't code, and config doesn't need a staged rollout. It does now.
- 04
Tale · 2026-08-06
The observability platform was down during the outage. Good. Now do it with your hands.
A junior engineer froze during a production fire because the SaaS dashboard that watches production was part of the fire. RodHat on the tools that were on every Unix box before the kid was born, the racket that sold competence back to us as a monthly invoice, and why the fire isn't out.
- 05
Tip · 2026-08-05
Your restart policy is deleting the crime scene
Automatic restarts keep services available, but an aggressive restart loop can erase the timing, logs, cores, and state needed to understand why a process failed. Preserve evidence before recovery becomes amnesia.
- 06
Tip · 2026-07-25
Process supervision is a 40-line problem. It has been solved since 1997.
You don't need a service manager with a DNS resolver in it to restart a daemon that died. daemontools, runit and s6 do supervision trees, log rotation and clean shutdown in a handful of files, and the design idea underneath them is worth stealing even if you never install one.
The syscalls underneath
Version introduced, sourced from the man page. Full cards in the syscall reference, or check your kernel.
| Interface | Since | What it does |
|---|---|---|
| flock(2) | 4.2BSD; Linux 2.0 | Apply or remove an advisory whole-file lock tied to an open file description. |
| pidfd_open(2) | Linux 5.3 | Get a file descriptor that refers to a process, immune to PID reuse. |
More console tips (3)
- Your cron job is running twice and that's why the numbers are wrong
- Read a wedged process with your hands, not a dashboard
- Your shell pipeline succeeded because the last command was polite
Rod's tales (14)
- The service was healthy and accepting no new connections
- The transaction counter hit two billion
- The load was forty and the box was idle
- The machine that forgot what year it was
- The process that wouldn't die
- The OOM killer was doing its job
- The pipeline succeeded. We found out from a customer.
- The alert had been firing for eight months and it was right the whole time
- Staging had its own everything, except the one thing that mattered
- The temporary NFS mount that ran production for six years
- The billing job ran twice for six weeks and the totals still balanced
- The microservice that should have been a function call, and the eighteen months it took anyone to say so
- The restore worked perfectly and we still lost eleven days
- The Friday deploy that ate my weekend, and whose fault it actually was (mine)
MOTD (5)
- Observability priced per gigabyte means you log least when you need it most
- Your backups are fine and one account suspension deletes all of them
- A major serverless platform's cold-start times regressed 40% and nobody outside the forums noticed
- Your AI pair programmer went down for four hours and so did half of Hacker News' composure
- Your build pipeline depends on a stranger's free tier and you found out on a Tuesday