The health check was green because it was checking the wrong damn thing

The dashboard was green for forty-three minutes while the service accepted no useful work.

Not “degraded.” Not “slow.” Dead in every way customers care about. Requests entered, waited, and failed. The write queue had stopped draining. The database connection pool was exhausted. The worker process responsible for committing jobs had wedged itself around a lock and was still technically alive, which turned out to be the only fact our health check knew how to verify.

GET /health returned 200 OK and the word healthy.

The application was healthy in the same sense that a corpse with an employee badge is still authorized to enter the building.

The endpoint checked whether the web process could lie

The handler looked roughly like this:

receive request
return 200

No database query. No queue observation. No check that workers were advancing. No age limit on pending jobs. No distinction between “the process has an event loop” and “the service can perform its purpose.”

It had been written years earlier for an orchestrator liveness probe. That use was defensible: answer a cheap request so the platform can tell whether the process is responsive enough to receive a termination signal or restart.

Then monitoring reused it.

Then the load balancer reused it.

Then the status page reused it.

One endpoint designed to answer one narrow question became the universal oracle for every operational question because it already existed and returned JSON.

This is how infrastructure acquires theology.

Liveness and readiness are different questions

A liveness check asks whether the process is stuck badly enough that restarting it is appropriate.

A readiness check asks whether this instance should receive traffic now.

A service-capability check asks whether the system, as a whole, can complete the thing users came for.

Those are not interchangeable.

A process may be live but not ready while it warms caches or waits for a required dependency. It may be ready to serve reads while writes are broken. It may accept HTTP requests while every asynchronous job is collecting dust in a queue. It may pass a synthetic login and still fail the customer path that uploads a file, publishes a message, or settles a transaction.

The wrong response to this complexity is one health endpoint that recursively interrogates every dependency until your monitoring system becomes the largest source of production traffic.

The other wrong response is return 200.

We had measured existence, not progress

The useful signal in this outage was not whether the worker process existed. It was whether the worker had completed anything recently.

The queue exposed a monotonically increasing completion counter and a timestamp for the last successful commit. Nobody alerted on either. We alerted on process count, CPU, memory, and HTTP health.

The wedged worker consumed a little CPU, held memory, answered management requests, and completed zero jobs. It looked beautifully normal in every graph except the one we had not built.

Progress signals are often better than presence signals:

• timestamp of the last successful job
• age of the oldest pending item
• rate of completed work versus accepted work
• successful write/read round trip through the real storage path
• backlog growth over time
• fraction of requests reaching the final business outcome

A process table tells you software is occupying space. It does not tell you the software is earning rent.

Dependency checks need boundaries

After the incident, someone proposed making /health query the database, queue, object store, identity provider, and two downstream APIs.

That would have converted a useless check into a distributed denial-of-service mechanism with a boolean return value.

Dependencies fail in different ways, and not every dependency failure should remove an instance from service. If the recommendation engine is unavailable, the storefront may still sell products. If the write database is unavailable, read-only traffic may remain useful. If one regional API is failing, routing all traffic away from every instance can make the healthy region fail too.

Health should expose components and capabilities, not pretend the system has one pulse.

Something like:

{
  "process": "live",
  "traffic": "ready",
  "read_path": "ok",
  "write_path": "failed",
  "queue_oldest_seconds": 1840,
  "last_job_completed_at": "2026-08-04T17:11:02Z"
}

The load balancer may care only about traffic. The pager should care about write_path, queue age, and the absence of completions. The public status page should translate those into customer impact instead of publishing the emotional state of a PID.

Synthetic checks must complete real work

The best check we added after the outage created a small synthetic job, let it traverse the normal queue and worker path, verified the resulting record, and removed it.

It ran slowly enough not to become workload, frequently enough to detect a stall, and with a unique marker so operators could separate synthetic traffic from customer data.

It also had a deadline.

A synthetic check without a deadline is merely another customer waiting forever.

The check did not replace metrics. It validated the whole path. Metrics explained which stage had failed. Logs supplied detail. Traces showed where time disappeared. Each tool answered a different question, which is more useful than making one endpoint cosplay as observability.

Restarting hid the design flaw

During the incident, restarting the worker cleared the lock and drained the queue. We nearly called that the fix.

A restart was recovery. The defect was that the system could stop making progress without declaring itself unready, without tripping a progress alert, and without failing a real end-to-end check.

Recovery procedures are not evidence that monitoring worked. Frequently they are evidence that humans noticed what monitoring did not.

The final changes were boring:

1. Separate liveness from readiness.
2. Publish explicit component and capability status.
3. Alert on queue age and absence of completed work.
4. Add a bounded end-to-end synthetic transaction.
5. Tie status-page messaging to customer capability, not process health.
6. Document which check drives restart, routing, paging, and public communication.

None of that required machine learning, service mesh astrology, or a conference talk.

It required asking what “healthy” was supposed to mean.

Our endpoint had answered: the web process can still say yes.

The customers had asked a different question.