$RodHat_
Console Tips

Read a wedged process with your hands, not a dashboard

Published by

Read a wedged process with your hands, not a dashboard
Photo: AI-generated — no human photographer / RodHat AI Cover

I ranted about the dashboard being down mid-outage. Here is the part you can screenshot: the actual walk. Something is wedged, the graphs are blank or lying, and you have a shell. That’s enough. It has been enough since before the graphs existed.

Start with the load average — it’s a queue, not a percentage

uptime

The three numbers are the run-queue length averaged over 1, 5, and 15 minutes: processes that are runnable plus processes stuck in uninterruptible sleep. On a 16-core box, a load of 16 is “full.” A load of 340 is not “busy” — it’s a pile-up. A high load with low CPU usage is the tell that things are blocked, not working. Now go find what’s blocking.

Find the stuck one by its state, not its CPU

ps -eo pid,stat,wchan:32,comm --sort=-stat | head

The stat column is the whole story. R runnable, S interruptible sleep (normal idle), D uninterruptible sleep, Z zombie, T stopped. D is the one that ruins afternoons: the process is blocked in the kernel — usually I/O or a lock — and it will not wake, will not be scheduled out cleanly, and will not die to kill -9. That’s not a bug in your kill. Uninterruptible means uninterruptible; the process has to return from whatever kernel call it’s in first. If you see D on a worker that should be flying, stop looking at dashboards.

Ask the kernel exactly what it’s waiting on

/proc is the kernel’s own confession, mounted as files. No tool required — just read them:

pid=$(pgrep -f my-worker)
cat /proc/$pid/wchan; echo          # the kernel function it's asleep in
grep State /proc/$pid/status        # State: D (disk sleep) etc.
sudo cat /proc/$pid/stack           # kernel stack (root; needs the symbol)
ls -l /proc/$pid/fd | tail          # what it has open right now

wchan gives you one word — futex_wait_queue, io_schedule, rwsem_down — and that one word usually names the whole problem. futex* means it’s stuck on a userspace lock (deadlock, or waiting on a peer that’s gone). io_schedule/blk* means storage. rwsem/mutex means kernel lock contention. /proc/$pid/fd shows the socket or file it’s parked on.

Watch it breathe with strace

sudo strace -p "$pid" -f -T -tt

If the process is alive but stuck, strace prints one syscall and then hangs there with it — that’s your answer: it’s blocked in that call. -T shows time-in-syscall, -tt timestamps, -f follows threads.

One caveat that trips people: on most modern boxes /proc/sys/kernel/yama/ptrace_scope is 1, so you can only attach to your own descendants unless you’re root or have CAP_SYS_PTRACE. Use sudo. If a process is in pure D state, strace may not attach at all — because there are no syscalls happening; it’s asleep in the kernel. That’s when wchan and /proc/$pid/stack are the better window.

The confession is usually already in dmesg

sudo dmesg -T | tail -50

-T gives human timestamps instead of seconds-since-boot. This is where the kernel already told you what happened and nobody read it: the OOM killer reaping your worker’s children, EXT4/XFS I/O errors, a NIC resetting, or the hung-task detector:

INFO: task my-worker:4131 blocked for more than 122 seconds.

That message is the incident. The kernel noticed the D-state stall two minutes in and wrote it down for free.

If it’s a JVM, ask it for a thread dump — don’t kill it

kill -QUIT "$pid"     # SIGQUIT → full thread dump to the JVM's stdout/log
# or:
jstack "$pid"

SIGQUIT to a JVM does not crash it — it dumps every thread’s stack to stdout. That’s how you find the two threads holding each other’s locks. (On a native process, SIGQUIT does core-dump; know which one you’ve got before you fire.)

See what it’s holding

sudo lsof -p "$pid"      # files, sockets, pipes this PID holds
sudo lsof -i :5432       # who's on the database port

A worker holding a connection that the pool thinks is free, or a deleted-but-still-open logfile eating your disk, shows up here in one line.

The boring operational version

  1. uptime — high load + low CPU = blocked, not busy.
  2. ps -eo pid,stat,wchan,comm — find the D (or the runaway R).
  3. cat /proc/<pid>/wchan + grep State /proc/<pid>/status — what it’s waiting on.
  4. strace -p <pid> if it’s making syscalls; /proc/<pid>/stack if it’s in pure D.
  5. dmesg -T | tail — the OOM killer / hung-task / I/O confession.
  6. kill -QUIT (JVM) or lsof -p for locks and held resources.
  7. Restart to recover — then fix the thing that let it wedge silently, or you’ll be back. (That last step has its own tip.)

None of this needs an agent, a license, or a working dashboard. It needs a shell and the willingness to read what the machine has been telling you the whole time. Same as pointing tcpdump at what’s actually on the wire: the truth is already there. Go take it.