Topic
Reliability & Observability
Production incidents, failure modes, observability, deployment safety, and resilient operations.
The pool was full of dead connections
An intermittent burst of broken-pipe database errors had been logged as "flaky" for two months before anyone looked closely enough to notice they were always the first query on a connection. The connection pool was handing out corpses.
`sudo` wrote its last heap overflow in C. The Rust rewrite shipped.
sudo-rs is production-ready for most deployments. A setuid-root binary sitting at uid 0, accumulating privilege-escalation CVEs for four decades, is exactly the right target for a memory-safe rewrite. This one is hard to argue with.
The OOM killer was doing its job
A slow memory leak ran undetected for five weeks because the kernel's out-of-memory killer, executing its heuristic correctly, kept choosing the monitoring agent over the leaking service. The pager never fired. The monitoring gaps were there in the data the whole time.
ss -i shows you what netstat never could: TCP internals live
Everyone knows ss -tulnp. Almost nobody uses ss -i, which surfaces congestion window size, RTT, retransmit counts, and send/receive buffer fill — directly from the kernel, no tool required.
The number 1024 was not a coincidence
A file descriptor limit from 2003 survived three infrastructure migrations and one complete platform rewrite because nobody ever questioned a number that had always been there. It bit us at 2am on a Tuesday.
The pipeline succeeded. We found out from a customer.
A deployment that broke production ran undetected for forty minutes because the notification step was written to never fail, and the monitoring alert that should have fired went to the same broken endpoint.
The postrotate script worked. We had no logs.
Three months of production logs, all gone. logrotate ran clean on schedule every week. The daemon was healthy and writing the whole time. The postrotate signal was firing correctly — just to a PID that had been dead since deployment day.
Your browser encrypted your DNS queries. It also decided who gets to read them.
DNS-over-HTTPS is a correct answer to a real problem — cleartext DNS has been leaking every hostname you visit since 1983. The way browsers deployed it, pointing users to Cloudflare and Google by default without explanation, is a different story. The encryption is real. The 'privacy' framing is doing a lot of work.
The storage was fine. We were swapping.
Three weeks, two engineers, one open storage vendor ticket, and roughly forty collective hours chasing read latency on a ZFS pool. The pool was fine the whole time. There was a swapfile nobody remembered adding seven months earlier.
The change was correct. Deploying it to every region simultaneously was not.
A one-line config change, reviewed, tested, and correct in every environment we tried it in. It went to all six regions in the same rollout because config isn't code, and config doesn't need a staged rollout. It does now.
The alert had been firing for eight months and it was right the whole time
A rule that pages nightly and gets acknowledged nightly isn't monitoring, it's a ritual. We had trained an entire team to dismiss a specific alert without reading it, and then it started telling us about something new.
Observability priced per gigabyte means you log least when you need it most
Ingest-based pricing creates an incentive that fires exactly backwards: the incident that generates the most telemetry is the one that costs the most to observe. Teams respond by sampling, and then wonder why the postmortem has a gap in the middle.
The observability platform was down during the outage. Good. Now do it with your hands.
A junior engineer froze during a production fire because the SaaS dashboard that watches production was part of the fire. RodHat on the tools that were on every Unix box before the kid was born, the racket that sold competence back to us as a monthly invoice, and why the fire isn't out.
Your restart policy is deleting the crime scene
Automatic restarts keep services available, but an aggressive restart loop can erase the timing, logs, cores, and state needed to understand why a process failed. Preserve evidence before recovery becomes amnesia.
The health check was green because it was checking the wrong damn thing
A service can answer HTTP 200 while its queue is wedged, its database writes are failing, and every useful request is dying. RodHat on health checks that prove process existence instead of service capability.
Stop waiting beside tcpdump like it owes you an incident
Intermittent network failures do not happen while you are watching. Use tcpdump's rotating capture files, size limits, and post-trigger preservation to keep the packets that existed before the pager fired.
Staging had its own everything, except the one thing that mattered
Separate servers, separate config, separate deploy pipeline, a big banner saying STAGING. And a database connection string that pointed, through two layers of indirection, at production. We found out during a load test.
The diff nobody wrote and everybody approved
AI coding tools now generate a meaningful percentage of commits at shops that have adopted them. RodHat on what happens to institutional knowledge when the author of the diff has never been paged.
We wrote the retry logic to survive a blip. It turned a blip into four hours.
One backend got slow for ninety seconds. Every client retried, in lockstep, three times, with no jitter. The retries were larger than the original traffic, the backend never recovered, and every fix we tried made it worse until we did the thing nobody wanted to do.
zfs send is the backup tool you already have and probably aren't using
An incremental zfs send moves exactly the blocks that changed, verified end to end, with no file-tree walk at all. The full replication pipeline is one command, and the resume token turns a failed 3TB transfer from a restart into a continuation.
The temporary NFS mount that ran production for six years
Somebody stood it up in an afternoon to unblock a launch. It had no monitoring, no owner, no backup, and no entry in any diagram. It survived two datacentre moves, a company acquisition, and every engineer who knew it existed.
Stop your backup job from taking production down with it
nice does nothing for I/O. ionice only works on one scheduler that most systems no longer use. cgroup v2 io.latency and io.max are the two knobs that actually work, and they work on the thing you already have.
The billing job ran twice for six weeks and the totals still balanced
No errors. No alerts. The reconciliation report matched every single day. And a few hundred customers were being double-charged, because the job that generated the charges and the job that verified them were the same code with the same bug.
Get a real disk latency histogram in one line of bpftrace
iostat gives you an average, and averages hide the tail that's actually hurting you. A four-line bpftrace program prints a log2 histogram of block I/O latency per device, live, on a production box, with no agent and no restart.
Passkeys fix phishing and nobody wants to talk about account recovery
The cryptography is right, the phishing resistance is real, and I've stopped arguing about it. What hasn't been solved is what happens when the device is gone — and every deployed answer to that reintroduces the thing passkeys replaced.
Your AI pair programmer went down for four hours and so did half of Hacker News' composure
A major AI coding assistant had a multi-hour outage this week. RodHat has seen this movie before, just with a different lead actor.
The migration was flawless. The TTL was 86400.
Six weeks of planning, a rehearsed cutover, and a maintenance window we finished forty minutes early. Then a fifth of our traffic kept arriving at a datacentre we'd already started decommissioning, for a full day, and there was nothing whatsoever we could do about it.
Process supervision is a 40-line problem. It has been solved since 1997.
You don't need a service manager with a DNS resolver in it to restart a daemon that died. daemontools, runit and s6 do supervision trees, log rotation and clean shutdown in a handful of files — and the design idea underneath them is worth stealing even if you never install one.
Certificate lifetimes are collapsing to 47 days and your renewal process is a person
The CA/Browser Forum voted to phase TLS certificate validity down to under seven weeks by 2029. If your renewal runbook has a human in it, the schedule just became your problem — and that's the actual point of the change.
The restore worked perfectly and we still lost eleven days
We tested our backups. Monthly, documented, signed off. The restore ran clean, the checksums matched, the database came up on the first try. And the data in it was eleven days old, because for eleven days the job had been backing up a directory nobody was writing to any more.
The Friday deploy that ate my weekend, and whose fault it actually was (mine)
A sysadmin war story about a Friday-afternoon deploy, a silent DNS TTL assumption, and the two-day outage it caused. RodHat owns every part of it.