The observability platform was down during the outage. Good. Now do it with your hands.

Production was on fire and the kid was staring at a spinner.

Not a graph. A spinner. The observability platform — the one we pay per-host, per-metric, per-custom-tag, per-deep-breath — had picked the exact same fifteen minutes as our outage to have an outage of its own. Their status page was yellow, which is vendor for “it’s red but our lawyers are awake.” So the dashboards that watch prod were themselves down, and a bright, expensive, well-meaning engineer was sitting in front of a loading animation like it owed him an answer, because the loading animation was the only interface to the machine he had ever been given.

I want to be clear that this is not the kid’s fault. This is our fault. We did this to him.

I leaned over and said: move. Not unkindly. Okay, a little unkindly. ssh to the box.

Ninety seconds with tools older than the engineer

uptime. Load average of 340 on a 16-core box. That number is not “busy.” That number is a run queue with a body count — processes piled up runnable-or-worse, waiting on something that is never coming.

ps -eo pid,stat,wchan:32,comm | grep ' D'. There it is. Our worker, state D. Uninterruptible sleep. The kid asked if we should kill -9 it. No, son. You can’t -9 a D. That’s the whole point of D. The kernel is holding that process underwater and won’t even let it die until whatever it’s blocked on returns. kill -9 is you politely asking. D is the machine saying not now, I’m busy lying to you.

cat /proc/$(pgrep -f worker)/wchan. The kernel tells you, in one word, the function it fell asleep in. strace -p on the same PID: one line, then nothing, hung in a futex. It wasn’t disk. It was a lock. A worker had grabbed a mutex and then wandered off to wait on a connection pool that had wandered off to wait on the worker. A deadlock. Older than me. Older than Usenet. Dijkstra was writing about this while your parents were in diapers.

dmesg -T for the confession — the OOM killer had been out working overtime, reaping children, and the pool never recovered. Restart the worker, the lock clears, the queue drains, the fire’s out. Elapsed time from ssh to green: about ninety seconds, and I spent thirty of them explaining what D meant.

No dashboard. No agent. No per-host license. procps, strace, and a kernel that has never once, in fifty years, refused to tell you the truth if you knew how to ask. /proc is right there. It has been right there the whole time. It is the machine’s own honest confession, mounted at a path, and we have raised a generation of engineers who have never cat’d a single file in it because we hid the machine behind a webpage and charged them rent to look through the keyhole.

They sold you the dashboard and kept the machine

Here’s the con, and it is a beautiful one, I almost respect it.

Somewhere in the last fifteen years the industry figured out that you could take the things every Unix box already does for free — strace, tcpdump, /proc, dmesg, lsof, a SIGQUIT to a JVM for a full thread dump that costs you exactly zero dollars — wrap them in a purple gradient, put them behind a login, and bill for them monthly. And it worked, because the same industry stopped teaching anyone what was under the gradient. You can’t miss what you were never shown. You’ll happily pay $180k a year to watch a metric you could cat out of a file, if nobody ever told you the file exists.

And it’s not just the tools. It’s r00t itself. Half the engineers I meet have never had it. Not on a lark, not on a Raspberry Pi, never. Their laptop is owned by an MDM agent that treats them as a threat. Their “server” is a container they cannot ssh into, scheduled by an orchestrator they cannot see, on a host that belongs to a company whose entire business model is making sure the machine is never, ever yours. You don’t own a computer anymore. You rent one back from the Gibson by the second, and they’ve convinced you that’s progress — that being locked out of your own box is a feature, a “managed experience,” a fucking value-add.

I’ll say the quiet part in the open, the way the Mentor said it in Phrack in ‘86 and nobody who needed to hear it did: this is our world now, the world of the electron and the switch. Except somewhere along the line we let them put a paywall on the switch and a EULA on the electron.

And yes — before the comments start — I still won’t touch systemd, and I still don’t think a service manager should have opinions about my DNS, my logins, and the weather. That’s a separate rant. I’ve got a folder.

The machine never stopped telling the truth

Here is the thing that should make you angry and then make you free: none of it was taken. It’s all still there.

The kernel still writes /proc/<pid>/status with the process state in plain ASCII. bpftrace will let you ask the running kernel questions in one line that we’d have killed for in 1999 — the modern kids who do go looking have better instruments than my whole generation ever had. dtrace, ktrace, strace, eBPF: four decades of people who refused to accept a boolean when they could have the truth. The vendors didn’t invent observability. They enclosed it, like a common, and sold tickets back to the field you already owned.

The dashboard being down didn’t blind us. It just removed the thing that had been standing between the kid and the machine his whole career, and for ninety seconds he got to see the machine directly, and — this is the part I actually came here to tell you —

He asked me to teach him

After it was green, after the retro, the kid didn’t say “we should buy the enterprise tier so this never happens again.” He said: “What was that wchan thing. Show me that.”

And there it is. That’s the whole fire, right there, still lit under a decade of SaaS invoices. Because you cannot kill it. You can bury it under abstractions, price it behind a login, MDM it off the laptop, and it will still flare up the instant some 24-year-old watches a D-state process refuse to die and feels that specific, ancient, chemical thrill of wait, what is it actually doing. That thrill is not nostalgia. It is not mine to hoard. It’s the same one Acid Burn had staring at the Gibson, the same one you had the first time a machine did exactly what you told it and nothing else and you understood, in your spine, that there was no magic — only layers, and every one of them would open if you learned its language.

There is no right and wrong here. There’s only fun and boring. And renting your competence back from a vendor, one dashboard at a time, until an outage in their datacenter can make you helpless in yours — that, my friends, is boring. It is the most boring possible way to be an engineer.

So here’s the homework, and it’s the same homework it’s always been. Next time the dashboard is up, ignore it. ssh to the box anyway. cat a file in /proc you’ve never read. strace something harmless and watch it breathe. Learn what your machine says when nobody’s translating. Own the layer under the one you were handed.

The platform will go down again. It always does. And when it does, be the one who leans over and says move.

Hack the planet. It’s still yours if you’ll just go touch it.