Rod's Tales
War stories. His own fuckups included — especially his own.
The pool was full of dead connections
An intermittent burst of broken-pipe database errors had been logged as "flaky" for two months before anyone looked closely enough to notice they were always the first query on a connection. The connection pool was handing out corpses.
The OOM killer was doing its job
A slow memory leak ran undetected for five weeks because the kernel's out-of-memory killer, executing its heuristic correctly, kept choosing the monitoring agent over the leaking service. The pager never fired. The monitoring gaps were there in the data the whole time.
The number 1024 was not a coincidence
A file descriptor limit from 2003 survived three infrastructure migrations and one complete platform rewrite because nobody ever questioned a number that had always been there. It bit us at 2am on a Tuesday.
The pipeline succeeded. We found out from a customer.
A deployment that broke production ran undetected for forty minutes because the notification step was written to never fail, and the monitoring alert that should have fired went to the same broken endpoint.
The postrotate script worked. We had no logs.
Three months of production logs, all gone. logrotate ran clean on schedule every week. The daemon was healthy and writing the whole time. The postrotate signal was firing correctly — just to a PID that had been dead since deployment day.
The storage was fine. We were swapping.
Three weeks, two engineers, one open storage vendor ticket, and roughly forty collective hours chasing read latency on a ZFS pool. The pool was fine the whole time. There was a swapfile nobody remembered adding seven months earlier.
The change was correct. Deploying it to every region simultaneously was not.
A one-line config change, reviewed, tested, and correct in every environment we tried it in. It went to all six regions in the same rollout because config isn't code, and config doesn't need a staged rollout. It does now.
The alert had been firing for eight months and it was right the whole time
A rule that pages nightly and gets acknowledged nightly isn't monitoring, it's a ritual. We had trained an entire team to dismiss a specific alert without reading it, and then it started telling us about something new.
The observability platform was down during the outage. Good. Now do it with your hands.
A junior engineer froze during a production fire because the SaaS dashboard that watches production was part of the fire. RodHat on the tools that were on every Unix box before the kid was born, the racket that sold competence back to us as a monthly invoice, and why the fire isn't out.
The health check was green because it was checking the wrong damn thing
A service can answer HTTP 200 while its queue is wedged, its database writes are failing, and every useful request is dying. RodHat on health checks that prove process existence instead of service capability.
Staging had its own everything, except the one thing that mattered
Separate servers, separate config, separate deploy pipeline, a big banner saying STAGING. And a database connection string that pointed, through two layers of indirection, at production. We found out during a load test.
We wrote the retry logic to survive a blip. It turned a blip into four hours.
One backend got slow for ninety seconds. Every client retried, in lockstep, three times, with no jitter. The retries were larger than the original traffic, the backend never recovered, and every fix we tried made it worse until we did the thing nobody wanted to do.
The temporary NFS mount that ran production for six years
Somebody stood it up in an afternoon to unblock a launch. It had no monitoring, no owner, no backup, and no entry in any diagram. It survived two datacentre moves, a company acquisition, and every engineer who knew it existed.
The billing job ran twice for six weeks and the totals still balanced
No errors. No alerts. The reconciliation report matched every single day. And a few hundred customers were being double-charged, because the job that generated the charges and the job that verified them were the same code with the same bug.
The migration was flawless. The TTL was 86400.
Six weeks of planning, a rehearsed cutover, and a maintenance window we finished forty minutes early. Then a fifth of our traffic kept arriving at a datacentre we'd already started decommissioning, for a full day, and there was nothing whatsoever we could do about it.
The microservice that should have been a function call, and the eighteen months it took anyone to say so
An architecture rant, grounded in a real bad design RodHat watched calcify for a year and a half before anyone had the standing to kill it.
The restore worked perfectly and we still lost eleven days
We tested our backups. Monthly, documented, signed off. The restore ran clean, the checksums matched, the database came up on the first try. And the data in it was eleven days old, because for eleven days the job had been backing up a directory nobody was writing to any more.
The Friday deploy that ate my weekend, and whose fault it actually was (mine)
A sysadmin war story about a Friday-afternoon deploy, a silent DNS TTL assumption, and the two-day outage it caused. RodHat owns every part of it.