The pipeline succeeded. We found out from a customer.

Forty minutes is a long time for a broken deployment to sit in production without anyone noticing. I know this because I was on call, I had a pager, and the pager was silent the whole time.

The pipeline

It was a decent pipeline. Tests, build, integration check, a smoke suite against staging, a deploy step with a manual confirmation gate, then a smoke suite against production. If the production smoke failed, the pipeline raised an alert and rollback was a button press. We’d built it carefully. It had caught real problems.

The last step — after the production smoke suite — posted a webhook to an internal notification service. The notification service dispatched status messages and alerts to the team’s chat. A successful deploy posted a green checkmark, the service name, git ref, who triggered it. A failing smoke suite posted a red X, the failing test name, a link to the log, and it simultaneously paged the on-call.

The notification step looked like this in the pipeline config:

notify --webhook "$NOTIFICATION_URL" --payload "$PAYLOAD" || true

The || true was intentional and documented. The comment read: notifications should not block deploys. The reasoning: if the notification service has a transient blip, you don’t want it to prevent a valid deploy from completing. The notification is informational. The smoke tests are the gate.

This is a reasonable position. It is also wrong, for reasons that only become clear when you are sitting in front of a broken production environment that the pipeline is reporting as fine.

The afternoon the tooling server filled up

The internal notification service ran on a shared tooling server. Tooling servers accumulate things — build artifacts, old coverage reports, log files from jobs nobody runs anymore — and this one had been accumulating for two years without a cleanup policy. The disk filled up at half past three in the afternoon, which is not three in the morning, which is what makes the story worse.

When the disk filled, the notification service could not write to its queue. It started returning 500s on every webhook call.

The pipeline noticed, in the sense that the notify step received a non-zero exit code. It noticed this by running || true, swallowing the failure, and continuing. The deploy completed. Production smoke ran. The production smoke suite posted a webhook to the notification service. That also failed, silently.

The smoke result — which was, and I want to be precise, a failure; the suite caught the regression — went nowhere. No chat message, no page, no alert. The pipeline row in the deployment history was green because the pipeline’s final state was the exit code of notify || true, which was zero.

The pager was quiet because the page was a webhook to the same service returning 500s.

Forty minutes

The regression was bad. Not catastrophic — no data loss, no cascade — but a significant slice of users were getting errors on a high-traffic flow. Under normal circumstances, you’d see that in chat within thirty seconds and have a rollback initiated in under two minutes.

We found out forty minutes later when a user posted in the company’s public support channel. Someone on the support team escalated it through a ticketing system — a different path entirely, not through the notification service — and it reached the right person, who looked at production metrics, said something I will not repeat here, and rolled back.

Post-rollback: check the pipeline. Everything green. Check the notification service: disk full, 500s everywhere, errors logged nowhere useful because the logging also tried to write to the same disk.

Check the smoke suite result from the failed deploy: it was there in the pipeline artifact store, correctly marked as a failure, waiting for someone to look at it. Nobody had looked because nobody had been told to look.

What had actually happened

Observability infrastructure that fails silently is not observability. This is that lesson wearing a different hat.

The || true was not wrong because it ignored notification failures. It was wrong because it treated the notification step as purely informational when the notification step was, in practice, the primary mechanism by which the on-call would know a deploy had gone wrong. We had conflated “the notification is not load-bearing in the execution of the deploy” with “the notification is not load-bearing.” It is not load-bearing in the first sense. It absolutely is in the second.

An alert that fires into a void is exactly as useful as an alert that never fires. The difference is that an alert that never fires at least doesn’t give you false confidence that the mechanism is working.

The tooling server disk fill was the proximate cause, and disk fills happen — that is the nature of disks. The question is always what the failure mode looks like from the operator’s seat when they do. Ours was: everything looks fine.

What we changed

The notification step lost its || true. Notification failures became pipeline failures. If the notification service is down, the pipeline stops with a clear error, the deploy does not complete, and the pipeline row is red. This is worse behaviour in the narrow case where the notification service blips during a healthy deploy; it is considerably better behaviour in the case where the notification service is down and the deploy broke production.

We added a fallback: if the primary notification path fails, the pipeline attempts a secondary path — email to a distribution list, bypassing the notification service entirely — and logs the attempt. If both fail, the pipeline fails loudly. The secondary path is boring and slow and it works when the interesting path doesn’t, which is the only property that matters when you need it.

The notification service got monitoring that didn’t depend on the notification service. A lightweight probe checked its health endpoint every sixty seconds and wrote results to a separate time-series store with its own alert path — not another webhook, a direct write. We called this “monitoring the monitoring,” which sounds like a recursion joke until it catches the disk-fill event at fourteen minutes instead of forty.

The tooling server got a disk cleanup cron and a usage alert at 70%. This is the most embarrassing fix because it’s the most obvious and it didn’t exist. The cron overlap guard is a ten-minute job; the actual problem was that the cleanup cron didn’t exist at all. Nobody owned the tooling server, which meant everyone assumed someone else was watching it. The classic tragedy of shared infrastructure.

The thing worth saying plainly

Your deploy pipeline is not just a build system. In most shops, it is the primary incident detection and response coordination mechanism — the thing that tells humans what is happening so humans can respond. The pieces that feel like “just notifications” are the parts that operators actually use to coordinate. They feel optional because the code got built and deployed without them. They are not optional in the sense that without them, nobody knows what is happening.

When you write || true on any step that touches your observability path, you are saying: this step is allowed to fail silently. Know what you are silencing. If you are silencing the silence-breaker — the thing that tells you when other things have gone wrong — then you have not added resilience to your pipeline. You have added a trapdoor.

The pipeline was working correctly. The smoke suite caught the regression. The failure was recorded, correctly, in the artifact store, waiting to be found. Nobody looked for forty minutes because nobody was told to look, and nobody was told to look because the thing that would have told them had been configured to fail quietly.

That’s a policy decision. It’s worth making on purpose.