The alert had been firing for eight months and it was right the whole time

The postmortem question was: how long had the system been telling us?

The answer was two hundred and forty-one days. Every night. To a channel with fourteen people in it. It had been acknowledged, by a human being, roughly seven hundred times.

The alert

DiskUsageHigh: /var/lib/app on batch-worker-03 above 85%

It first fired on a Tuesday in the autumn. Somebody looked, found a temp directory that hadn’t been cleaned up, deleted some files, and the alert cleared. Normal operations, no ticket, thirty seconds of work.

It came back four days later. Somebody deleted some more files. Then it started coming back nightly, at around 3am, when a batch job ran and produced intermediate output that wasn’t cleaned up properly.

Somebody wrote a cron job to clear the directory at 4am. And so a pattern established itself: the alert fires at 3-something, whoever’s on call acknowledges it, the cleanup runs at 4, the alert clears. Every night.

Nobody fixed the batch job. There was a ticket. It was low priority, because there was a workaround, and the workaround worked. I’d have deprioritised it too.

Eight months of training

Here’s what actually happened over those eight months, and it wasn’t a monitoring failure. It was a training programme.

Every night, an alert fired. Every night it was safely ignorable. Fourteen people, night after night, learned — not consciously, not as a decision anyone made — that a message beginning DiskUsageHigh on a batch worker is noise.

By month three, people were acknowledging it from their phones without opening it. The shape of the notification was enough. By month six, one of the on-call engineers had a phone filter that auto-dismissed anything matching that string, which they mentioned in the postmortem with visible embarrassment and which I think was an entirely rational response to the situation we’d put them in.

We hadn’t just failed to fix a problem. We had systematically conditioned a team to not read a specific class of message, and then we left that channel connected to the thing that would eventually need to tell us something.

What changed

In month eight, the batch job’s input volume grew past a threshold, and the intermediate output stopped being cleaned up in a single 4am pass. The cleanup started leaving a residue. Small at first — a few gigabytes a night.

The alert kept firing at 3am. It kept getting acknowledged. But now it wasn’t clearing at 4am; it was clearing at 4:20, then 5:00, then not fully clearing at all, with the baseline creeping up night after night.

The alert was, during this entire period, correct. It was telling us the disk was filling up. It was telling us with increasing urgency, in the sense that it was staying on for longer each night. And the information “this alert is now behaving differently than it did last month” was not something the alerting system could express and not something a human acknowledging a notification at 3am on their phone was ever going to notice.

The disk filled completely on a Sunday. The batch worker was where the nightly reconciliation ran. Reconciliation failed silently — it wrote its output to the full disk, got an error, logged it, and exited zero, because somebody’s error handling had a bare except in it and that’s a different story.

Nobody looked at reconciliation output on Sundays. It failed again Monday, and Monday’s failure was noticed on Tuesday when finance asked why they hadn’t received a report.

Three days of unreconciled transactions, on a system where reconciliation was the mechanism by which we caught payment mismatches. Sorting that out took a week.

The postmortem

I want to describe the tone of it, because it went somewhere I didn’t expect.

The initial framing was alert fatigue — a known phenomenon, a known fix, tune your alerts, everyone nods. There was a moment where it was going to be written up that way and closed, with an action item about reviewing noisy alerts, which would have been done once and never again.

What changed it was somebody asking a much better question: how many other alerts are in this state right now?

Nobody knew. So we went and looked, and that was a genuinely bad afternoon.

We pulled twelve months of alert history and computed, for each rule, how often it fired and how often the acknowledgement was followed by any action — a commit, a ticket, a config change, anything.

Out of roughly two hundred active rules, eleven accounted for over eighty percent of all alerts fired, and of those eleven, nine had essentially no correlated action, ever. They fired, they were acknowledged, nothing happened, they cleared.

Nine rules were generating the overwhelming majority of the noise the on-call team experienced, and none of them had produced a single useful outcome in a year.

Meanwhile, forty-odd rules had never fired at all — which sounds fine and mostly wasn’t, because when we tested a sample of them by inducing the condition, six didn’t work. Wrong metric name after a refactor, a threshold that could never be crossed, a rule pointed at a service that had been renamed. They had never fired because they couldn’t, and their silence was indistinguishable from health.

The rules we adopted

An alert must be actionable, and “actionable” has a specific meaning. There is a documented action, it can be taken by the person receiving the alert, and it must be taken now rather than in the morning. If any of those is false it isn’t a page. It’s a ticket, or a dashboard, or nothing.

Anything firing more than once a week without an action is deleted or fixed. Deleted is an acceptable outcome and we had to say that explicitly, because there’s a strong instinct that removing monitoring makes you less safe. A rule everyone has been trained to ignore is not providing safety. It’s providing noise, plus a false sense that the thing is watched, plus active harm to every other alert that shares the channel.

Symptom, not cause. “Disk 85% full” is a cause and it may or may not matter. “Reconciliation has not completed successfully in 26 hours” is a symptom, and it matters every single time it’s true. The second one would have fired on Sunday morning, to a channel nobody had been conditioned to ignore, about the thing we actually cared about.

That reframing is most of the value. We had dozens of alerts about resource utilisation and almost none about whether the things the business depends on had actually happened.

Every rule gets tested. Induce the condition in a test environment, confirm the alert fires and reaches a human. Annually, and after any refactor of the thing it watches. Six of ours were decorative and we would never have known.

Track acknowledgement-to-action ratio as a first-class metric. Per rule. If a rule is acknowledged fifty times and correlates with zero changes, that rule is not monitoring anything. That number is now on a dashboard and it gets reviewed quarterly, and reviewing it is somebody’s named responsibility.

What I actually think about it

The instinct after an incident is always to add monitoring. Something got missed, so watch more things. Every postmortem I’ve been in has produced at least one action item that adds an alert.

This one produced a net reduction of about sixty rules, and the on-call experience improved so much that people commented on it unprompted within a fortnight.

We didn’t have a monitoring gap. We had monitoring that was, in aggregate, teaching a team of competent people to not look at their notifications — and we’d built that training programme ourselves, one deprioritised ticket at a time, over eight months, while congratulating ourselves on having good observability coverage.

The alert was right for two hundred and forty-one days. It never got quieter, it never got louder, and by the time it was describing an actual emergency it was speaking to a room where everyone had learned it never was.