A process is in state D, ps says it is “sleeping”, and kill -9 does nothing. On Linux you cat /proc/PID/stack. On FreeBSD there is no /proc you should be relying on (it is not mounted by default and has been deprecated for years), and the answer is a tool most people have never typed: procstat(1).
It reads the same kernel data through sysctl, needs no mounted filesystem, and answers the questions you would otherwise burn ktrace on. I covered the tracers in ktrace, kdump and truss. This is the cheaper tier: look at the process, do not trace it.
Where is it stuck: kernel stacks
procstat -kk 4121
One -k prints the kernel stack as function names. Two -kk add the offsets, which is what you paste into a bug report. Each thread gets a line per frame, with the wait channel on the first one:
PID TID COMM TDNAME KSTACK
4121 100412 nfsd - mi_switch sleepq_switch sleepq_catch_signals ...
If every thread of a hung daemon is parked in the same nfs, zfs or vnode frame, you have just learned which subsystem is the problem without attaching anything. Run it twice, ten seconds apart. A stack that changed is a slow process. A stack that is identical is a wedged one.
You generally need root for this. Ordinary users get less, or nothing, for other users’ processes.
What does it have open: -f
procstat -f 4121
Roughly the lsof -p answer, without installing lsof. You get the fd number, the type (vnode, socket, pipe), the access flags, the offset, and the path where the kernel can resolve one. Sockets show protocol and the local and remote addresses. A daemon with 60,000 descriptors, 59,900 of them to one peer in CLOSE_WAIT, tells you whose bug it is. For the whole-box version of this question see what is actually listening.
Add -a to walk every process:
procstat -af | awk '$4 == "v" && $NF ~ /deleted-file-name/'
Adjust the match to taste. The point is that -a plus awk is the poor man’s “who is holding this file open”, which is the usual end of a full disk with no files story.
What is it mapped: -v
procstat -v 4121
The virtual memory map: start and end addresses, protection (rw-, r-x), resident page count, reference and shadow counts, the type of backing object (vnode, swap, device, default) and the path. Use it when RSS looks wrong. A thousand anonymous rw- regions of 64 MB each is an arena allocator that never gave anything back. One huge rw- region with a swap type is the process the OOM killer will eventually pick.
Everything else worth knowing
-tlists threads with their state, CPU, priority and wait channel. First stop for “one core pinned, rest idle”.-ris resource usage (user and system time, page faults, context switches).procstat -r -w 5 4121repeats every five seconds so you can watch the counters move. A process whose voluntary context switches climb while nothing else does is waiting on something.-lshows its resource limits, which is useful after setting rctl limits on a jail and wondering why a daemon died.-cprints the command line arguments and-ethe environment, as the process actually has them, not as the init script claims.-bprints the path of the binary. This catches the daemon still running a deleted executable after an upgrade.-iand-jshow which signals are pending, ignored or caught, and-sthe credentials. A process ignoringSIGTERMexplains a lot.
Flag sets move a little between releases. Read procstat(1) on the machine in front of you before you script against column positions.
The cons
- It is a snapshot. A process that is wedged for 200 ms every minute needs
ktraceor dtrace, not a luckyprocstat. - Kernel stacks and other users’ details need privilege, and
security.bsd.see_other_uidsand jail settings can hide processes entirely. - Output is built for people. Parsing columns with
awkworks until a path contains a space. -von a process with hundreds of thousands of mappings is slow and large. Redirect it to a file.- It tells you where a process is, not why. A kernel stack ending in a lock wait still needs the holder found.
Stack first with -kk, files with -f, memory with -v, and only then reach for a tracer. The FreeBSD and OpenBSD hub has the rest of the toolbox.
