$RodHat_
MOTD

Your build pipeline depends on a stranger's free tier and you found out on a Tuesday

Published by

Your build pipeline depends on a stranger's free tier and you found out on a Tuesday
Photo: AI-generated — no human photographer / RodHat AI Cover

ERROR: toomanyrequests: You have reached your pull rate limit.

CI goes red across every project at once. Nothing was deployed, nothing was changed, no code moved. You are being rate limited by a registry you don’t pay, from a shared NAT address you don’t control, because a hundred other tenants on the same CI runners spent your anonymous quota before your job started.

Then it clears up and everyone goes back to work, having learned nothing.

The thing nobody wants to say

Free container registries were never going to stay free at unlimited volume, and everybody knew it. Serving petabytes of layer data to anonymous CI jobs is a real cost, paid in bandwidth, borne by a company with shareholders. The surprising part isn’t that limits arrived. It’s that anyone built a production pipeline that assumed they wouldn’t.

And the same shape shows up everywhere else: package registries, language-specific mirrors, GitHub raw content, the base OS image mirrors your Dockerfile pulls from. Every one of them is a free service on somebody else’s balance sheet that your deploys structurally depend on.

You didn’t choose that dependency. It accumulated. Somebody wrote FROM node:20 in 2021 and nobody has thought about it since.

The fix is old and boring

Run a pull-through cache. It has existed the whole time.

# registry config
proxy:
  remoteurl: https://registry-1.docker.io

Point your CI at it. Now the first pull of a layer goes upstream and every subsequent pull for the next six months is served from your own disk. One registry account’s quota covers your entire organization because you make one request per unique layer rather than one per build.

You also get: builds that don’t fail when the upstream registry has an outage, faster CI because the layers are on your network, an audit trail of every image your builds have ever pulled, and the ability to keep shipping during whatever the next pricing change is.

Harbor, Artifactory, Nexus, or the plain upstream registry:2 image with eight lines of config. This is an afternoon of work. The reason it doesn’t get done is that it has no owner — it’s not a feature, it’s not an incident, it’s infrastructure hygiene that only becomes visible in the twenty minutes a quarter when everything is broken and everyone is too busy to fix the cause.

Pin your bases while you’re in there

FROM node:20 is not a version. It’s a moving pointer that changes under you — same tag, different bytes, different CVEs, different behavior — and it means your build from last Tuesday is not reproducible today.

FROM node:20.11.1-bookworm@sha256:a3f2...

The digest is the actual identity of the image. The tag is a label somebody can repoint. If you care that your build produces the same output twice, pin the digest and update it deliberately with a bot, like any other dependency.

Yes, that means an update PR every week. That’s the correct amount of friction for changing the base of everything you ship.

Where I was wrong

I spent years arguing that the answer was to stop depending on public registries entirely — build your own base images from a distro you control, and own the whole chain.

That’s still right for a certain class of shop, and I’d defend it for anyone with a compliance regime or a real security team. But I underrated the cost. Somebody has to maintain those base images, track upstream CVEs, rebuild on a schedule, and be responsible when the hand-rolled base is three months behind on a libc fix. For most teams that’s a worse security posture than pulling a well-maintained upstream image through a cache, because the upstream image gets patched by people whose entire job it is and yours gets patched when someone remembers.

The cache is the right answer for almost everyone. It gets you the availability and the quota, without signing up for a maintenance burden you’ll abandon in eight months.

The general lesson, which is not about containers

Go make a list of every external service your deploy pipeline hits between “merge” and “running in production.” Actually make it — read the CI config, read the Dockerfiles, read the install scripts.

For each one: who pays for it, what happens when it’s down, and what happens when it changes its terms. If the answer to any of those is “I don’t know” or “the build fails,” that’s not a dependency, it’s a bet — and you’re going to find out how it resolves on a Tuesday, at the worst possible moment, along with everyone else who made the same bet.