The Friday deploy that ate my weekend, and whose fault it actually was (mine)

I want to get ahead of the obvious joke: yes, I deployed on a Friday. I know. I knew at the time, which is worse. This isn’t a story about someone else’s dumb mistake dressed up as a teaching moment — it’s mine, and I still think about it more than is probably healthy.

The setup

Small shop, maybe eight engineers, I was the closest thing to an SRE team the place had. We were migrating a customer-facing API off an aging box onto new hardware — same application, new IP, and because I’d done this migration a dozen times before at other places, I was confident. Confidence is the tell, in hindsight. Every story like this has a moment where the narrator was confident.

The plan was simple: cut the DNS TTL down days in advance so the eventual switch would propagate fast, bring up the new box, verify it was healthy, flip the A record, watch traffic move over, decommission the old box the following week once nothing was pointing at it anymore. Textbook.

Where it actually went wrong

I lowered the TTL. I verified the new box. I flipped the record. Traffic moved over cleanly within about fifteen minutes, which felt like vindication. I decommissioned the old box’s application that same Friday afternoon — stopped the service, left the hardware up, told myself I’d do the full teardown Monday. Standard stuff. I went home for the weekend feeling good about myself, which is the exact feeling you should learn to distrust.

What I hadn’t accounted for: a chunk of our traffic — not from browsers, from a handful of B2B integration partners hitting the API directly — was coming through resolvers that were, for reasons that took me two days to fully understand, ignoring the TTL I’d set and caching the old record for significantly longer. Some of it was a misconfigured caching layer on the partner side. Some of it, I eventually learned, was a resolver at a regional ISP that was known — known to people who weren’t me, at the time — for TTL-flooring behavior under load, silently extending short TTLs to something it considered more “reasonable” to reduce query volume upstream.

I had stopped the service on the old box. The partners still resolving to the old IP got connection refused, immediately, all weekend, with no fallback and no alert firing on my end because from my monitoring’s perspective — watching the new box — everything was green. The outage was invisible to me and total for a subset of people who mattered a great deal to the business.

What actually fixed it, eventually

A partner’s on-call engineer emailed our sales contact Saturday afternoon, sales forwarded it Sunday night, and I found out my “clean migration” had been silently broken for over 40 hours when I opened my laptop Monday morning to do the teardown I’d deferred. I brought the old application back up on the old box within twenty minutes of finding out — that part was easy, I hadn’t torn down the hardware. The damage was already fully done by then; two days is two days.

What I actually changed after this

Not “never deploy on a Friday,” though sure, fine, that too. The actual lesson, the one that changed how I do every migration since:

• TTL is a request, not a guarantee. Some resolvers respect it faithfully. Some don’t, for reasons ranging from misconfiguration to deliberate cache-flooring policy you’ll never get visibility into. Plan the cutover assuming a meaningful fraction of clients won’t honor your TTL on schedule — because they won’t, and you have no way to know in advance which ones.
• Never decommission the old path until you’ve actively confirmed nothing’s still using it, not “waited a period of time I assumed was safe.” Leave both paths serving traffic and watch access logs on the old box specifically for a full week before you touch it. If the old box still shows incoming connections after your TTL window “should” have expired, that’s not noise, that’s your answer.
• Silence in your monitoring is not the same as health. My dashboards were green because they were watching the box that was working. They had nothing to say about the box I’d already stopped caring about — which is exactly where the actual damage was happening the whole time.

I’ve run probably thirty migrations since that one and every single one of them keeps the old service running, unloved and mostly idle, for a full week past when I’m sure I don’t need it anymore. It costs nothing but a little discomfort at “why is this still up.” That discomfort is cheaper than another Saturday spent finding out from a forwarded email that your confidence was premature.

Also: the reason I could reconstruct any of this after the fact is that Backblaze had been quietly doing its job the whole time. Get backups you don’t have to think about, so the worst part of an incident is the incident and not also discovering your backup strategy was theoretical.