The process that wouldn't die

There are a handful of things in Unix that you believe without examining. Not faith, just accumulated experience: signals work that way, the kernel does this, the shell does that. You’ve seen it a thousand times. The belief is so old it’s invisible.

kill -9 kills the process. That’s one of them.

The signal number 9 is SIGKILL, and SIGKILL cannot be caught, blocked, or ignored. A process has no say in whether it lives once SIGKILL arrives. This is by design, documented in every Unix manual that has ever existed, and confirmed by every operator who has ever had to put down a hung process. You send kill -9, the process is gone. That’s the deal.

The morning the stuck process didn’t die was the morning I stopped treating that as settled.

What the batch job was doing

The job was reading input files from an NFS mount and transforming them. Nothing exotic. It had been running for years without incident. This particular run had started at 03:14, and by 09:30 someone noticed it still had not completed and had produced no output in the last four-plus hours.

The process was there in ps, consuming no CPU, holding file handles, doing nothing. I sent SIGTERM first, which is the polite way, and then waited ten seconds and sent it again. Nothing happened.

kill 14823
kill 14823

The process was still there.

kill -15 explicitly. Nothing. Then kill -9.

kill -9 14823

I ran ps aux | grep 14823. The process was present. Same PID, same start time, still running as root, no change. Not a zombie, not <defunct>. A live process that had just received SIGKILL and was ignoring it.

I sent kill -9 four more times. Not because I thought repetition would help, but because my brain needed to exhaust the obvious before it would accept that something unusual was happening. The process sat there.

The STAT column

root     14823  0.0  0.0  12840  2304 ?        D    03:14   0:00 /usr/bin/batch-job --input /mnt/nfs/data/run-20260903

That D in the STAT column. That is the whole story.

Most processes you’ll ever look at are in S state: sleeping, interruptible. They’re blocked in a select() or poll() or nanosleep(), waiting for something to happen. When a signal arrives, the kernel can wake them from the sleep, check the signal, and dispatch it. SIGKILL arrives, the process does not get a choice, the kernel terminates it. This is the behavior you’ve seen ten thousand times.

D state is different. D stands for uninterruptible sleep. The process is not waiting in userspace for an event. It is inside a kernel function, executing kernel code, and that kernel code cannot be interrupted right now. The kernel is not checking for pending signals. Your SIGKILL has been queued as pending. It will not be delivered. The kernel code path will not reach a signal-check point until the operation it’s performing completes, and the operation is not completing.

SIGKILL cannot kill a D-state process. SIGKILL and SIGSTOP are the only two signals that cannot be masked by a process using sigprocmask(), but that is an entirely different constraint. The issue here is not that the signal is masked. It is that signal delivery requires the process to be at a dispatch point, and the kernel code the process is running in will not yield to a dispatch point until it finishes what it’s doing.

If the operation never finishes, the signal never gets delivered. The process never dies.

Why the process was in D state

The input path was /mnt/nfs/data/run-20260903. The NFS server that served /mnt/nfs had rebooted the previous night after an unplanned power event. Our NFS client had not noticed. The server was back up, but it had come back with a fresh NFS stack and had dropped all its previous session state. The client, still believing the session was valid, sent a read RPC to the server. The server returned an ESTALE error: stale file handle. The client did not handle this gracefully.

Instead, the NFS client code went into a retry loop inside the kernel’s NFS read path. It was waiting for the read operation to succeed. The operation was not going to succeed because the file handle was stale and the client wasn’t doing anything useful to recover. The retry was persistent, configured by the mount options.

mount | grep nfs
nfsserver:/data on /mnt/nfs type nfs (rw,relatime,vers=3,...,hard,proto=tcp,timeo=600,retrans=2,...)

hard. That’s the option that made this an unbounded problem.

A hard NFS mount retries indefinitely. It does not give up. It does not return an error to the calling process. The design intent is correct: if you’re writing to shared storage that matters, you don’t want a transient network hiccup to silently return EIO to your application and potentially corrupt data. Hard mounts will keep trying until the server comes back.

But if the server comes back with dropped state and the client doesn’t recover gracefully, “keep trying until it works” becomes “keep trying forever.” The process doing the read is in D state for as long as the kernel NFS retry loop continues. Which, with a hard mount, is until something external intervenes.

The fix

umount -f /mnt/nfs on a hung hard NFS mount. Force unmount. On Linux, if the mount is in active use, force unmount alone may not be enough:

umount -l /mnt/nfs

The -l flag is lazy unmount. It detaches the mount from the VFS name tree immediately, so no new opens can reach it, while allowing existing file handles to drain. The NFS code receives an error from the kernel, the retry loop stops, the D-state process hits a dispatch point, SIGKILL is finally delivered, and the process exits.

After umount -l, the batch job was gone within a few seconds. The whole thing had taken about ninety minutes of confusion, ten minutes of actual diagnosis once I understood what I was looking at.

What you need to know about D state going forward

D-state processes come from a handful of causes. NFS hangs with hard mounts are the most common thing you’ll see on a production server. But the pattern is the same regardless of source: the process is inside kernel code that will not return until the kernel operation succeeds or the kernel subsystem is forced to give up.

Other things that produce D state:

A disk or disk controller goes unresponsive. The kernel retries the block I/O. Any process doing I/O through that device is stuck until the hardware recovers or the system is rebooted. There is no software fix. If the disk is genuinely dead, you reboot.

A faulty kernel module can put a process into D state if the module has a bug in its wait path. Diagnosing this requires looking at what the process is actually waiting on, which you can do with cat /proc/<pid>/wchan, which prints the kernel function name the process is sleeping in. A more complete walkthrough of reading a wedged process through /proc is in the tips section.

Brief D states during heavy I/O are normal. A process doing disk reads will pass through D state transiently as individual I/O operations complete. The problem is D state that lasts minutes, not microseconds.

# find D-state processes
ps aux | awk '$8 ~ /^D/ {print $0}'

# see what kernel function a specific PID is sleeping in
cat /proc/14823/wchan

If the wchan output points at an NFS function and you have NFS mounts, check your server. If it points at an I/O function, check your disks. If it points at something you don’t recognize, that’s the kernel module to investigate.

The NFS mount option argument

hard versus soft NFS mounts is an old argument and I’ll give you both sides honestly.

soft mounts return EIO to the application after a configured number of retries fail. The application sees an error and can handle it, or not. This means D-state hangs are unlikely. It also means that if you’re writing to an NFS mount and the network flickers at the wrong moment, your write may fail silently from the application’s perspective, and depending on how well your application handles EIO, you may get silently incomplete or corrupted data. For read-only data, scratch space, or batch jobs where failure is recoverable, soft is reasonable. For anything where write durability matters, soft mounts are a latent data-integrity problem.

hard mounts do not lose writes to transient failures. They also make every process doing I/O through that mount hostage to the server’s availability. The right response to this is not to switch to soft mounts. The right response is to know that this is how it works, mount with intr if your kernel version respects it (many modern kernels don’t anymore), and have umount -l in your runbook for when a server drops state and doesn’t come back clean.

The NFS mount that “temporarily” unblocked a launch and then ran production for six years is a different failure mode, but it comes from the same underlying pattern: NFS dependencies that nobody has thought through end-to-end before something goes wrong.

The batch job in this incident was running on a hard mount for the right reasons: its input data was important, partial reads would have produced garbage output, and we wanted the job to wait for the server rather than fail silently. What was missing was a runbook entry covering “what do you do when the job gets stuck and kill -9 does nothing.” That entry now exists. It says: check STAT in ps, look for D, check wchan, check whether any open files are on NFS, run umount -l, then SIGKILL.

The thing that takes the longest to rebuild

Every operator learns, eventually, that signals can be masked. You read about sigprocmask(), you understand that a process can block most signals, you file that away. What most people don’t hold clearly is the distinction between blocking a signal and being unable to reach signal delivery.

SIGKILL cannot be masked. That’s true. But signal delivery still requires a dispatch point, and a process in D state doesn’t reach one until the kernel finishes whatever it started. “Cannot be masked” and “will always be delivered promptly” are different guarantees, and Unix only offers the first one.

The process was not ignoring SIGKILL. It was not caught in a signal handler. It was not a zombie. It was doing work inside the kernel, work that had no intention of completing, and until we took the NFS mount away from underneath it, there was nothing at the process level that was going to change that.

kill -9 is not the final word. The kernel is.

Once you know this, the diagnosis is five minutes: ps, STAT column, D, cat /proc/pid/wchan, NFS path in lsof -p. The first time, without this in your head, you’ll spend an hour checking that you’re actually root. The same kind of invisible kernel ceiling showed up in the conntrack table incident: iptables accepted the packets, the kernel dropped them anyway, and nothing at the userspace level explained why.

When the userspace tool stops working, go one layer down. The kernel usually has a clear answer. It just doesn’t announce itself unless you ask.