A major serverless platform's cold-start times regressed 40% and nobody outside the forums noticed

Buried in a community forum thread with maybe forty replies: cold-start latency on one of the big serverless platforms crept up roughly 40% over the last month, measured independently by three separate people running their own synthetic benchmarks because the vendor’s own dashboard, shocker, doesn’t surface this cleanly.

No incident report. No status page entry: this wasn’t an outage, it was a slow bleed, and slow bleeds don’t trigger the automated systems that make vendors say things publicly. It just quietly got worse, and unless you’re the specific kind of person who benchmarks their own cold starts on a schedule, you wouldn’t know.

The thing about serverless that everyone forgets

The pitch was always “stop thinking about the machine.” Fine, good pitch, mostly delivered on. But “stop thinking about the machine” quietly became “stop measuring the machine” for a lot of teams, and those are not the same thing. You outsourced the ops burden, not the responsibility for knowing whether your p99 latency is where you think it is.

I ran my own boxes for twenty years before any of this existed, and the one habit that carried over cleanly into the serverless era is: trust nobody’s dashboard for anything that affects your SLA. Vendor dashboards are built to make the vendor look good in aggregate, not to catch a 40% regression that only shows up in the tail.

What to actually watch

• Run your own synthetic cold-start probe, on a schedule, outside the platform’s own tooling. Doesn’t need to be fancy: a scheduled invocation and a timestamp diff gets you 90% of the value.
• Alert on p99, not average. A 40% regression concentrated in the tail is invisible in a mean and extremely visible to the unlucky user who hit it.
• If you’re multi-cloud or multi-region for anything that matters, compare cold starts across providers periodically. Relative drift is often easier to spot than absolute drift.

Nobody’s going to page you for this. It’s not a breaking change, it’s not a security bulletin, it’s just a platform getting slightly worse in a way that’s easy to miss and annoying to prove. Which means it’s exactly the kind of regression that sits there for months unless somebody’s actually checking, and increasingly, in this industry, nobody is.

The pattern I’ve seen since 1997, restated for the current decade

The specific technology changes. The dynamic doesn’t. I watched the same thing happen with shared hosting in 2001: providers would pack more customers onto a box over time, your site got slower, there was no announcement, no SLA breach, nothing to point at. “It’s just slower now” is hard to document and harder to escalate. The vendor knows this.

With serverless the wrinkle is that the vendor’s own aggregate metrics look fine, because a 40% cold-start regression is invisible in an average when cold starts are a small fraction of total invocations. Your warm-path p50 latency is probably unchanged. The regression shows up in the tail, and in new deployments, and in the functions you haven’t warmed lately. The places you check least often.

The forum thread that caught this regression had timestamps. The first person to notice posted a data point and assumed they’d misconfigured something. The second person, two weeks later, asked if anyone else had seen it. The third person ran a proper before/after comparison. Total time from first symptom to community confirmation: about three weeks. Meanwhile every function on that platform had been running slower for a month.

If you’re running a scheduled synthetic probe on a separate account with no other activity, you catch this in the first window. If you’re relying on whether anyone mentions it in a forum, you’re three weeks behind on a regression that’s already in production.

The bpftrace IO latency tip is overkill for this specific case, but the underlying habit, instrument first and trust the measurement over the dashboard, transfers directly.