The migration was flawless. The TTL was 86400.

The migration itself was the best-run piece of work I’ve been part of. I want to say that first, because everything after this paragraph is about how it went wrong, and it wasn’t the plan that was bad.

Six weeks of preparation. Full rehearsal in a staging environment that genuinely matched production for once. A runbook with rollback steps at every stage, reviewed by three people, with named owners and expected durations per step. A war room, a comms plan, a status page draft written in advance so nobody had to compose prose at 2am.

We finished forty minutes ahead of schedule. Somebody ordered food. It was, for about ninety minutes, the most professionally satisfying night of my year.

What we’d moved

Old colocation facility to a new provider. Not a lift-and-shift into cloud — a physical move to better hardware in a better building with a better contract, planned around a lease that was expiring and would not be renewed on terms anybody could defend.

Everything replicated ahead of time. Databases in sync, application servers built and burned in, load tested against synthetic traffic. The cutover itself was, deliberately, almost nothing: verify final sync, stop writes at the old site, confirm replication caught up, flip DNS, verify, resume.

The DNS flip was one line in the runbook, right at the end, marked as low risk. Which it was, in the sense that the change itself was trivial and worked immediately.

4am

I was watching the new site’s graphs come up and looking, out of habit, at the old site’s too.

The old site’s traffic dropped. Not to zero. It dropped to about a fifth of normal and then flattened out into a line, dead level, no downward trend at all.

I stared at that line for a long time before I understood what I was looking at, and when I did the food stopped being appealing.

Twenty percent of our traffic was still arriving at a facility we had just stopped writing to, and it was going to keep arriving there. The application servers at the old site were still up — we hadn’t turned them off, because the runbook sensibly said to leave them warm for rollback — and they were still serving requests, against a database we had deliberately made read-only and stale.

Users were getting a working site. It just didn’t have anything they’d done in the last two hours in it.

The number

I pulled the zone. The A record’s TTL was 86400.

Twenty-four hours. Somebody had set it years earlier, entirely reasonably — the record hadn’t changed in the whole time it had existed, and a long TTL on a stable record is good hygiene. It reduces query load, it makes you more resilient to a DNS outage, it’s what you’re supposed to do.

Nobody lowered it before the migration. It wasn’t in the runbook. Six weeks of planning, three reviewers, a full rehearsal — and the rehearsal used different hostnames in a different zone, so the TTL never came up, because in staging nothing had been cached anywhere for a year.

Resolvers all over the world had our old address, with up to twenty-four hours left on it, and there is no mechanism by which you can reach into a stranger’s resolver and tell it to forget. None. You cannot push a DNS invalidation. That is not a feature that exists.

What “propagation” actually means

I’ve heard people say “DNS is propagating” my entire career and I’ve said it myself and it’s a lie that costs people money.

Nothing propagates. Your zone is authoritative from the moment you save it — dig +trace would have shown the new address immediately, and it did, which is why the first person to check said everything was fine. But every resolver that had already asked kept the old answer until its copy expired. And every client behind those resolvers kept getting the old answer, and every one of those clients was doing real work against a read-only replica of a database, and none of them had any way to know.

There’s no propagation delay. There’s a cache expiry, and its length is a number you chose, in advance, possibly years ago, possibly not by you.

The hour we spent inventing options

We considered a lot of things between 4am and 5am, and I’ll list them because the process of eliminating them is where the real lesson is.

Turn off the old site. Now that twenty percent of users get connection refused instead of stale data. Is a hard failure better than a soft one? Genuinely arguable — at least it’s visible — but it converts “some users see old data” into “some users see nothing at all,” for up to a day, and the retries would hammer us.

Proxy from old to new. This is what we did, eventually, and it’s the right answer. Reconfigure the old site’s load balancers to forward everything to the new site’s public address. Users resolving to the old IP get correctly served, just with an extra hop and some latency. Took about ninety minutes to do safely, mostly spent making sure we weren’t creating a loop.

Move the old IPs. Not possible; different provider, different address space. This is worth knowing in advance — if you’re moving within a provider and can bring the addresses, the entire problem evaporates.

Wait it out. Twenty-four hours of split-brain, with users doing work that would be silently discarded. Not acceptable, and the fact that it was even discussed tells you how few options we had.

The proxy worked. We kept the old site’s front end alive for eight days, watching the traffic decay — and it decayed in the shape you’d expect, a fast drop as the well-behaved resolvers expired, then a long, infuriating tail from resolvers ignoring TTLs, clients caching at the application layer, and at least one large corporate network that appeared to have hardcoded our address in something.

Eight days. On a twenty-four hour TTL.

What I’d have done differently

The fix is embarrassingly simple and it has to happen before the maintenance window is even scheduled:

Two weeks out, drop the TTL to 300. And understand why the timing matters: the change to the TTL is itself subject to the old TTL. If the record has been at 86400 for a year, resolvers holding it will keep holding it for up to a day before they pick up the shorter value. You must lower it, then wait a full old-TTL period, before the low TTL is universally in effect.

That single step turns a twenty-four hour tail into five minutes and it costs nothing but a calendar entry.

Put the TTL check in the runbook as a prerequisite, not a step. With a verification command and an expected output. dig +noall +answer and read the number with your own eyes.

Rehearse against the real zone. Our rehearsal was excellent and it tested everything except the one thing that broke, because staging had no cache history. Anything that behaves differently by virtue of having existed for a long time cannot be rehearsed in an environment that was created last month.

Have the proxy ready before you need it. We invented it under pressure at 4am. It should have been a prepared, tested step in the runbook labelled “if stale resolution persists” — thirty minutes of work in advance instead of ninety under stress.

The thing that stayed with me

We planned for six weeks and the failure was a number nobody looked at, on a record nobody had touched in years, set correctly by someone doing the right thing at the time.

That’s most incidents, in my experience. Not a bad decision — a stale one. Something that was correct when it was set and became a landmine when the surrounding facts changed, sitting there for years being quietly right until the day it wasn’t.

The runbook had forty-one steps. It should have had forty-two, and the missing one was fourteen days earlier.