Staging had its own everything, except the one thing that mattered

Somebody asked me once what the most frightening thing I’d ever seen on a terminal was. It’s this, and it’s not close:

staging=> SELECT count(*) FROM users;
  count
---------
 2841097

Staging had about four hundred test users. It had four hundred test users that morning. I was looking at production, from a shell I had opened specifically because it was not production, in the middle of a load test we had deliberately designed to be destructive.

The environment

It was a good staging environment, which is the part that makes this worth telling. Not a neglected one.

Separate hosts, separately provisioned. Separate load balancer, separate DNS zone, separate deployment pipeline with its own credentials. The application rendered a bright orange banner across the top of every page saying STAGING so nobody could confuse the two in a browser. There was a documented refresh process that copied a sanitised subset of production data in weekly.

Somebody had clearly cared about this. It had all the visible markers of a properly isolated environment.

The indirection

The database connection didn’t come from a config file. It came from a config management layer, which pulled from a secrets store, keyed by environment name, which was derived from a host attribute, which was set at provisioning time.

Four layers between “what database does this connect to” and any file a human might read. Each layer was individually defensible. Config management so hosts are reproducible. A secrets store so credentials aren’t in the repo. Environment-keyed lookup so the same artifact deploys everywhere. Host attributes so provisioning is automated.

Together they meant that answering the question “what database is staging actually talking to” required tracing through four systems, three of which needed different credentials to query, and none of which displayed the resolved final value anywhere.

So nobody ever checked. Not out of laziness — because checking was a twenty-minute job requiring access most people didn’t have, and the answer was obviously “the staging one.”

How it broke

Three weeks earlier, staging had been rebuilt. Hardware refresh, entirely routine.

The provisioning script set the host attribute environment=staging. The secrets store had entries keyed by environment. But somewhere in the rebuild, the new hosts came up before the staging entry existed in the secrets store for that particular key path — a race between two automated processes that had never overlapped before.

And the lookup had a fallback. Of course it did. Someone had added it eighteen months earlier to handle a transient failure that was causing deploys to fail intermittently, and the fallback was: if no environment-specific value is found, use the default.

The default was production. Because the default had been written when production was the only environment.

The staging hosts came up, failed to find a staging database entry, fell back to the default, connected to production, and worked perfectly. Better than usual, in fact — somebody remarked that week that staging felt unusually snappy and had realistic-looking data. It did. It had exactly the same data as production, because it was production.

Nobody investigated why staging felt good. Nobody ever investigates why something feels good.

Three weeks

For three weeks, everything deployed to staging ran against the production database.

We reconstructed what happened during those weeks and it was, by a margin I still find hard to accept, mostly fine. Reads, overwhelmingly. Some test users created — in production, where they sat looking like real accounts. One data migration dry-run that a developer had run to check timing, which had modified about two hundred rows before they cancelled it, and they’d assumed the odd behaviour they saw was staging data being weird.

That developer knew something was wrong. They saw something inconsistent with their expectations and rationalised it, because “staging is a bit odd” is a completely normal thing to believe about a staging environment. That’s the piece that haunts me — the system gave a warning to a competent person and the warning was indistinguishable from ordinary staging weirdness.

Then somebody scheduled a load test.

The load test

It was a serious one. Ten times expected peak, sustained for an hour, with a workload mix that included writes — account creation, order placement, the whole path. Precisely the kind of test you can only run in an environment you’re willing to destroy, which is why we were running it in staging.

Someone opened a psql session against the staging database to watch table sizes during the run. That’s where the count came from. That’s the number at the top of this article, seen about ninety seconds before the test was due to start.

I have never seen a room move that fast. The test was cancelled before it began. Nothing was lost. We got extraordinarily lucky, and I want to be precise about what the luck was: it was that somebody idly opened a database session to watch a graph. There was no control that caught this. A person was curious.

What was actually wrong

Not the fallback, though we removed it. Not the race, though we fixed it.

The actual problem was that the system had no way to state what it was connected to. Four layers of indirection resolved to a value at runtime, and that value appeared nowhere — not in a log line, not in a health endpoint, not on the staging banner that had been carefully built to prevent exactly this class of confusion.

The banner said STAGING because a config value said staging. It was reporting its intent, not its state. Every marker of isolation we had was declarative — this is meant to be staging — and none of it was observational.

That’s the general failure, and it’s everywhere once you start looking. Systems that tell you what they were configured to be rather than what they turned out to be.

What we changed

The startup log line. On boot, every service logs the resolved database host, port, and database name — the actual values, after all indirection, from the live connection. Not the config; the connection. Somebody objected that it was noise. It is the single most useful line in our logs.

A health endpoint that reports identity. /healthz returns the environment name and the database it’s connected to, read from the connection rather than from config. Monitoring asserts these agree. If a host claiming to be staging is connected to a database named production, it alarms immediately and loudly.

Database-side enforcement. This is the good one. The production database now has a role for each environment, and the staging credentials are simply not valid against it. Not a policy — an authentication failure. If a misconfigured host tries to connect to production with staging credentials, it doesn’t fall back gracefully, it fails to start, which is the correct behaviour and which no amount of application-layer care can guarantee.

Fallbacks fail closed. Every default in the config resolution chain was audited. Any lookup that can’t find an environment-specific value now errors. A deploy that fails because a secret is missing is an inconvenience. A deploy that silently uses production is an outage waiting for a load test.

The banner reads the connection. It renders the database hostname it is genuinely talking to, in the banner, on every page. It’s ugly. It has prevented two further incidents that I know of, both of which were caught by somebody glancing at a page and going “hang on.”

The thing to take away

Every environment marker we had was a claim about intent. The hostname was staging-app-01 because someone named it that. The banner said STAGING because a variable said so. The deploy pipeline was called staging. The DNS zone was staging.

None of them were derived from what the system was actually doing, and the one thing that mattered — the database at the other end of the socket — was invisible behind four layers of well-intentioned abstraction.

Make your systems report their state, not their intent. Then make something check the two agree. Because if the only thing separating your test environment from your production data is a config value resolving correctly through four systems, then it isn’t separated. It just hasn’t diverged yet.