We wrote the retry logic to survive a blip. It turned a blip into four hours.

The initial fault lasted about ninety seconds. The incident lasted four hours and eleven minutes, and every single minute after the first ninety seconds was caused by our own code doing exactly what we had told it to do.

The ninety seconds

An authentication service — internal, called by nearly everything, cached aggressively — had a garbage collection pause turn into a series of garbage collection pauses after a deployment shifted its memory profile. For about a minute and a half it went from a 4ms p50 to timing out.

That should have been survivable. It was a well-understood dependency, everything that called it had timeouts configured, and the caches meant most requests never reached it at all. On paper, a ninety-second blip in that service degrades a few percent of requests and then it’s over.

What actually happened

Every service that called auth had retry logic. Sensible-looking retry logic, reviewed and approved, written independently by six different teams and looking remarkably similar because everyone had read the same blog posts:

retries: 3
backoff: exponential
initial: 100ms

Three attempts. 100ms, 200ms, 400ms. Nothing wrong with any individual line of that.

Now do the arithmetic across the fleet.

Auth starts timing out. Every in-flight request fails. Every one of them retries — so the load on auth is now roughly four times the original, since each request became one attempt plus three retries. Auth was already unable to keep up; now it’s receiving four times as much.

And because the backoff had no jitter, every client that failed at the same moment retried at the same moment. The traffic didn’t just quadruple, it arrived in synchronised waves — a spike at t+100ms, a bigger one at t+300ms, a bigger one at t+700ms — with the fleet marching in step because they’d all started their clocks on the same triggering event.

Then the second-order effects. The services calling auth started timing out to their callers. Those callers had their own retry logic. Retries of retries. A single user request at the edge was, by the time it reached the bottom of the stack, potentially sixty-four attempts.

Auth’s GC pause resolved after ninety seconds. It never got a chance to recover, because by then it was receiving something like fifteen times its normal traffic in synchronised bursts, from a fleet that had collectively decided to try harder.

The forty minutes of making it worse

I want to walk through what we tried, because the sequence is instructive and it is not flattering.

We restarted auth. Reasonable — clear the GC state, come up fresh. What actually happened: it came up with cold caches and empty connection pools, into fifteen times normal load, and fell over in about four seconds. We did this three times before someone said stop.

We scaled auth up. Doubled the instance count. The new instances came up, joined the load balancer, immediately received a share of the storm, and started timing out too. We had added capacity to a system that was failing because of a feedback loop, and a feedback loop does not care how much capacity you have — it scales to fill whatever you provide. All we’d done was increase the load on the shared database behind auth, which then started to struggle, which was genuinely novel and unwelcome.

We increased the timeouts. Somebody’s theory was that requests were timing out just barely, and a bit more patience would let them through. What this actually did was hold connections open longer, exhaust the connection pools, and increase the number of requests in flight at any moment. Measurably worse within two minutes.

Forty minutes in, we had restarted the service, doubled its capacity, and relaxed its timeouts, and it was in a worse state than when we started. Every one of those actions was defensible in isolation. Every one of them added energy to a system that needed energy removed.

What worked

Somebody who had seen this before — and this is the whole reason having such a person in the room matters — said: stop trying to serve the traffic. Turn the retries off.

That is not an easy thing to say out loud in an incident, because it sounds like giving up.

We had a feature flag for retry behaviour, which is the only piece of preparation in this entire story that paid off. We set retries to zero fleet-wide.

Load on auth dropped by roughly three quarters within seconds. It stabilised in under a minute. Then we brought retries back — one attempt, with jitter, in a staged rollout across the fleet — and watched to make sure the herd didn’t re-form.

Total time from “turn off the retries” to recovered: about six minutes. We’d spent three and a half hours before that.

The failing requests during those six minutes failed fast and cleanly. Users got errors instead of thirty-second hangs, which is a better experience by every measure and also generates far less load. That’s the counterintuitive part and it’s the real lesson: shedding load is how you recover, and serving fewer requests successfully beats attempting all of them unsuccessfully.

What we changed

Jitter, everywhere, non-negotiable. Full jitter, meaning the delay is a random value between zero and the exponential ceiling, not the ceiling plus a random smear. Without jitter, synchronised clients stay synchronised forever — the exponential backoff spreads them out in time but keeps them in phase, which is the worst of both.

A retry budget, not a retry count. This is the change that mattered most and it’s the least widely adopted. A count is per-request and therefore unbounded in aggregate: N requests failing means N × retries of extra load, with no ceiling. A budget is per-client, expressed as a fraction of successful traffic — retries may not exceed, say, 10% of successful requests in a rolling window. When the backend is healthy, that’s plenty. When everything is failing, the budget is exhausted almost immediately and retries stop automatically, without a human deciding to turn them off.

That property is the entire point. The system sheds its own load without anyone in a war room having to be brave.

Circuit breakers with actual half-open behaviour. After a threshold of failures, stop calling entirely for a period, then let a single request through to test. Not all of them at once — that’s just the thundering herd with a delay, and I have seen breakers implemented that way, which is worse than no breaker at all because it produces regular synchronised waves.

No retries below the top layer. This was the contentious one. If every layer retries, the multiplication is exponential in depth. We picked one layer — the edge — and made everything below it fail fast. Some teams disliked losing their local resilience. They were right that they’d lost something; they were wrong about what it was worth.

Load-test the recovery, not just the load. Our load tests proved auth handled peak traffic. They had never tested auth coming up cold into a fleet mid-retry-storm, which is the state it’s actually in when it matters. Now there’s a chaos test that does exactly that, and the first time we ran it, it failed.

The thing worth stealing

Retry logic is written by an engineer thinking about one request. From that seat, retrying is obviously correct — the request failed, the failure might be transient, try again. Every code review of that logic is conducted from the same seat.

Nobody is sitting in the seat where you can see all the requests at once, and from that seat, a fleet-wide retry policy is a positive feedback loop with a gain factor you chose without noticing you were choosing it.

The question to ask in review is not “should this retry.” It’s: if every caller of this service does this simultaneously, what multiple of normal traffic does the service receive, and can it survive that while cold?

If nobody in the room can answer, you don’t have retry logic. You have an amplifier pointed at your own infrastructure, and it’s armed.