The disk is full and du says it isn't. It's an open file descriptor.

df says the filesystem is at 100%. du -sh /* adds up to a third of that. Somebody is about to start deleting things at random.

Don’t. You already know what this is.

Why the numbers disagree

unlink(2) removes a directory entry. It does not free anything. The kernel frees the inode and its blocks when the link count hits zero and no process holds the file open. Delete a 40GB log while your application still has the fd, and the name is gone — so du, which walks names, cannot see it — but every block is still allocated, and df, which asks the filesystem for free blocks, absolutely can.

This is the single most common “disk full” incident in production, and it’s almost always the same story: log rotation ran, the config didn’t tell the daemon to reopen its logs, and the daemon has been happily writing to a file that has no name for eleven days.

Find it

lsof +L1

+L1 means “files with a link count less than 1” — deleted, still open. That’s it, that’s the command.

COMMAND   PID  USER   FD   TYPE DEVICE     SIZE/OFF NLINK  NODE NAME
java     2417  app    97w  REG  253,1   42949672960     0 12583 /var/log/app/app.log (deleted)

There’s your 40GB. PID 2417, fd 97.

Without lsof — a stripped container, a box where you can’t install anything:

ls -l /proc/*/fd 2>/dev/null | grep '(deleted)'

Same information, uglier. On FreeBSD, fstat gives you the equivalent view.

Fix it without a maintenance window

The obvious fix is restarting the process, which closes the fd and instantly returns the space. Sometimes you can’t — it’s the database, it’s midday, restarting it is the outage.

So truncate the file through the fd instead. The name is gone, but /proc/<pid>/fd/<n> is still a path to it:

: > /proc/2417/fd/97

Space comes back immediately, process never notices, no restart. Caveat worth knowing: if the process opened the file without O_APPEND, its file offset is unchanged, so the next write lands at byte 42949672960 and you get a sparse file that reports huge and occupies nothing. Cosmetically weird, operationally harmless. With O_APPEND — which any sane logger uses — writes go back to the start and you’re clean.

That trick has bought me an afternoon more than once.

Then stop it happening again

Truncating is triage, not a fix. The daemon is still writing to an unnamed file and will refill the disk on the same schedule. The actual bug is in your rotation config:

• logrotate — use copytruncate for daemons that can’t reopen, or a postrotate that sends the right signal. Most daemons reopen on SIGHUP or SIGUSR1; check yours, don’t guess.
• newsyslog on BSD — the signal column in /etc/newsyslog.conf exists precisely for this and half the entries people write leave it empty.
• Anything writing to a log path it opened at startup and never reopens is a landmine with a calendar.

The other one: the mount you buried

Second-most-common version of the same confusion, and it’s the reverse.

Mount a filesystem over /data while /data already has 200GB of files in it. Those files still exist on the underlying filesystem. They are unreachable — the mount covers them — so du /data reports what’s on the mounted filesystem, but the root filesystem is still carrying the buried 200GB and df / shows it.

Find it without unmounting anything:

mkdir /mnt/peek
mount --bind / /mnt/peek     # Linux
du -sh /mnt/peek/data
umount /mnt/peek

A bind mount of / shows you the root filesystem without anything mounted on top of it. There’s your buried directory, exactly where somebody left it before the storage migration in 2023.

The habit

When df and du disagree, they’re not broken and neither is the filesystem. One of them is counting names and the other is counting blocks, and the gap between those two numbers is always a story: a deleted file somebody’s still holding, or a directory somebody mounted over. lsof +L1 and a bind mount of / answer both in under a minute, which is a lot faster than the alternative — a stressed engineer deleting things they don’t recognize on a production box at 2am.