The change was correct. Deploying it to every region simultaneously was not.

Every organisation has a moment where it discovers that its config management system is a deployment pipeline with no safety features, and that it has been treating a mechanism capable of changing every server on earth in nine seconds as though it were a text editor.

Ours cost us fifty-one minutes of global downtime. Total, complete, every region, no degraded mode. The kind of outage where the status page is the only thing you own that still works.

The change

One line. A connection pool size, raised from 50 to 200 on a service that had been showing pool exhaustion under peak load.

It was correct. I want to be unambiguous about that, because it would be a tidier story if the change had been wrong. It wasn’t. The analysis was sound, the number was well-chosen, it had been running in the development environment for a week and in staging under load test for two days, and it did exactly what it was supposed to do.

It was reviewed by two people. It went through the normal process. And the normal process for config, unlike the normal process for code, applied it everywhere at once.

Why config was different

Our code deployments were properly done. Canary at 1%, bake for twenty minutes with automated metric comparison, then 10%, then 50%, then full, region by region, with automatic rollback on error-rate regression. Somebody had built that carefully and it had caught real problems.

Config went through a different system. Version controlled, reviewed, applied by a converge run that ran every fifteen minutes across the fleet. All of it. Every region, every host, within one converge cycle.

The reasoning, which was never written down but which everyone shared, was that config isn’t code. It’s data. It’s a number. The risky thing is the code that reads the number, and that code was already deployed and tested.

That reasoning is wrong and it’s wrong in a way that’s very hard to see from inside, because ninety-nine times out of a hundred it produces no problems at all, and the hundredth time it produces all of the problems simultaneously.

Fifty-one minutes

Pool size 200, applied to every instance of a service running a few hundred replicas across six regions.

Each replica opened up to 200 connections to the shared database cluster instead of up to 50. Under normal load they wouldn’t all be used — but connection pools are eager on startup in this particular library, and the converge run restarted the service to pick up the config.

So a few hundred replicas restarted within one converge window and each immediately opened its pool.

The database’s max_connections was, and this is the detail that turns a capacity problem into an outage, a number that the new fleet-wide total comfortably exceeded. The database started refusing connections. Refusing connections is not slow — it’s immediate and total. Services couldn’t reach the database at all. Health checks, which included a database ping, started failing. The orchestrator saw unhealthy replicas and restarted them. Restarted replicas opened their pools again, and again got refused.

Every region. At the same time. With no region left healthy to absorb traffic, because the thing that made them independent — separate replicas, separate load balancers, separate everything — did not extend to the config that had just been applied identically to all of them.

The rollback that didn’t

We identified the change in about six minutes, which I’m still proud of. Reverting it took forty-five, and this is the part I actually want to talk about.

The revert commit had to converge. Fifteen-minute cycle. We could force a converge run, and did, except:

The converge agent talked to a control plane that talked to the database. Not the same database, but one that had its own dependency on the auth service, which had its own connection pool to the cluster that was refusing connections. Our tool for changing configuration on the fleet was, in a way nobody had ever mapped, downstream of the thing we had just broken.

So we went to manual. SSH to hosts, edit the file, restart the service. Which works fine for one host and is an enormous problem for several hundred across six regions, especially since the connection reuse that would have made that bearable wasn’t configured on the jump hosts.

And the restarts had to be staggered. Because if we brought them all back at once with the corrected config, they’d all open their pools at once, and even at 50 connections each, several hundred replicas starting simultaneously against a database that was already refusing connections would have failed again. We had to bring regions back one at a time, slowly, watching connection counts.

Fifty-one minutes, most of it spent doing carefully by hand what the automation was supposed to do, using tools that were partly broken by the outage.

The postmortem’s uncomfortable finding

The change was correct. The review was correct. The testing was appropriate. Nobody made a mistake, in the sense of doing something they shouldn’t have.

What we had was a blast radius problem, and blast radius is a property of the delivery mechanism, not of the change. We had spent two years building excellent progressive delivery for code and had not noticed that the other pipeline — the one that could change the behaviour of every process in the company in one converge cycle — had none of it.

The second finding was worse. Nobody had mapped the dependency from the config management control plane to the database. It was three hops and it had never mattered, because the control plane had never needed to fix something the database depended on. Our remediation tool was inside the blast radius of the thing it was meant to remediate, and we discovered this while using it.

What we built

Config gets staged rollout, same as code. One region, bake, metrics compared automatically, then the rest. It made config changes take four hours instead of fifteen minutes, and there was real resistance to that — config changes are often how you fix an incident, and slowing them down has a cost.

So there’s a documented break-glass path for immediate global application, which requires a second approver and posts loudly in an incident channel. It gets used a few times a year and every use is reviewed. The point isn’t to prevent global changes. It’s to make choosing a global change a decision somebody makes deliberately rather than the default that happens because nobody specified otherwise.

Resource limits are validated against downstream capacity. A check in CI: if a config change raises a per-instance connection pool, multiply by the maximum replica count and compare against the database’s max_connections. Fails the build if it doesn’t fit. This is a laughably simple check and it would have caught this exact incident in review, before a human ever looked at it.

The control plane got a dependency-free path. The config management system can now be driven from a bastion with no dependency on any production service. It’s uglier and slower and it works when everything else doesn’t, which is the only time you need it.

Pool sizing got a floor and a ceiling as a policy, not a value. The real bug underneath all of this was that per-instance limits were being set without anyone holding the fleet-wide total in their head. Nobody’s job was to know that number. Now the number is in a dashboard and the CI check reads from it.

The thing worth remembering

We had progressive delivery, canary analysis, automatic rollback, and region isolation. We had, genuinely, good deployment practice — better than most places I’ve worked.

All of it applied to code, and none of it applied to the mechanism that could change the behaviour of every running process in the company in under a minute. Because config isn’t code, and nobody had ever said that out loud clearly enough for anyone to notice it was false.

Go and find every mechanism in your organisation that can change production. Not just the deploy pipeline — the config management, the feature flags, the DNS, the load balancer rules, the database migrations, the certificate rotation, the secret store. For each one, ask what fraction of production it can affect in one action, and how long it takes to undo.

The one with the widest blast radius and the weakest controls is where your next total outage comes from. It’s very rarely the deploy pipeline. That one has everybody’s attention.