The disk had space and nothing would write.

3am. Pager. “Application is down, writes are failing, disk full.”

I had been on call for this machine for three years. I knew the disks. There was no way they were full. I checked anyway.

$ df -h /var
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3        80G   38G   40G  49% /var

Forty gigabytes free. Not full. I went back to the alert. “No space left on device.” Back to df. Still forty gigabytes free.

Okay. Either something is broken in a way I have not seen before, or something is broken in a way I have definitely seen before and am blanking on at 3am. Both are possible. I stared at the terminal and tried to think.

The filesystem has two resource pools

This is the thing they do not teach you, and the thing that will burn you at the worst moment. A filesystem does not have one “space” resource. It has two.

The first is block space: the raw bytes available to store file data. This is what df shows. This is what everyone watches. This is what monitoring graphs. This is what page-7-of-every-sysadmin-course covers.

The second is inodes: the metadata structures that track file existence, ownership, permissions, and location. Each file requires one inode. The number of inodes on an ext2/ext3/ext4 filesystem is fixed at format time. If you run out of inodes, you cannot create new files, regardless of how much block space remains. The error you get is “No space left on device.” Same error. No distinction. No hint.

The command you want:

$ df -i /var
Filesystem      Inodes  IUsed   IFree IUse% Mounted on
/dev/sda3      5242880 5242879       1  100% /var

One inode remaining. Out of five million. The disk had forty gigabytes free and was completely full in every way that mattered.

Five million inodes is a lot until it is not

Ext4 defaults to one inode per 16,384 bytes of filesystem space. On an 80GB filesystem that sounds generous. You end up with around five million inodes. Five million distinct files. Who could possibly need more than five million files?

The PHP session directory.

The application was a web service that had been running for four years. PHP sessions. Default session handler, default session storage: /var/lib/php/sessions/. Default garbage collection: session.gc_probability = 1, session.gc_divisor = 1000. That means on each request there is a 0.1% chance the GC runs and cleans up expired sessions.

At low traffic, that is fine. The GC runs often enough to keep the session directory clean.

At high traffic it falls apart. The GC still runs on 0.1% of requests, but now “0.1% of requests” means thousands of GC runs per hour. Each GC run takes a few milliseconds. But GC runs are also not transactional. If a request gets killed mid-GC, you get half-cleaned sessions. Then a cron job someone added three years ago to “help” the GC along had a bug: it found sessions correctly, but wrote a completion marker for each one before deleting it. The marker files never got cleaned up.

Each session: one marker file. Each marker file: 12 bytes, but one inode.

Four years of traffic. Call it conservatively 500 requests per second at peak, sessions lasting 24 hours, session files persisting for 48 hours before cleanup. The math does not matter. What matters is that /var/lib/php/sessions/ contained 4.9 million files when I found it. Most of them marker files. Most of them 12 bytes.

Four and a half gigabytes of tiny files eating every inode the filesystem had.

Fixing the immediate problem

Deleting five million files from a single directory is not as simple as rm -rf /var/lib/php/sessions/*. Shell glob expansion runs out of argument space. rm -rf /var/lib/php/sessions/ tries to read the entire directory into memory. On a directory with millions of entries, that is a long time.

The right tool:

find /var/lib/php/sessions/ -maxdepth 1 -type f -mtime +2 -delete

find with -delete processes entries one at a time without building a giant in-memory list. On a slow disk with millions of inodes this will still take a while. It ran for forty minutes. The application came back up after about ten minutes once enough inodes were free.

Kill the cron job with the bug. Fix the PHP session config to use session.gc_probability = 10 and session.gc_divisor = 100. Fix the cron to stop leaving marker files everywhere. Add df -i to the disk monitoring so this never surprises anyone again.

The part where I complain about ext4

The inode count on ext4 is baked in at mkfs time. You can check it with tune2fs -l /dev/sda3 | grep "Inode count". You cannot change it on a live filesystem. You cannot add more inodes without reformatting.

If you know your workload creates enormous numbers of small files, you can pass -i bytes-per-inode to mkfs.ext4 and set a lower ratio. -i 4096 gives you one inode per 4KB instead of one per 16KB, roughly four times as many. You pay for this in a slightly larger inode table, slightly more overhead at mount time, and marginally worse directory lookup performance in some pathological cases.

You make this decision once, at format time, before you know what the disk will be used for, with imperfect knowledge of your future workload. This is obviously terrible fucking design.

FreeBSD’s UFS2 has had dynamic inode allocation since version 5.0. The filesystem grows its inode table as needed. You do not think about it. You do not have this problem. The filesystem is twenty years old. This was the correct call.

xfs does the same thing. Inodes are allocated dynamically from a pool that can grow within the filesystem’s space. You can technically exhaust it, but only if you have filled every block with inode metadata, which requires a truly pathological workload. In practice it does not happen.

btrfs also does dynamic allocation.

So the Linux ecosystem, after watching ext4 burn people with this exact problem for fifteen years, moved to filesystems that do not have the problem. Fair enough. I begrudgingly admit that xfs on a workload with variable file counts is the correct choice and I should stop telling people to use ext4 out of habit. There. I said it.

What you should actually do

Add df -i to your monitoring. Alert at 80% inode usage. This is it. This is the whole lesson. Everything else in this post follows from the 3am alert that could have been a 10am ticket.

This is the same category of failure as the daemon that ran with umask 077 for six months: a resource that looked fine by every metric you were watching, broken in a way you were not watching. The kernel also lies about memory in a similar fashion if you let it; madv_free and RSS accounting will gaslight you just as efficiently as df will here.

If you are formatting a new filesystem and you know it will hold many small files: use xfs or btrfs, or use mkfs.ext4 -i 4096 and accept the tradeoff.

If you are already on ext4 with a full inode table and cannot reformat: add a second mount point on a new disk specifically for the high-inode-count directory. Mount /var/lib/php/sessions/ on its own ext4 filesystem formatted with -i 4096, or on xfs.

The problem is completely avoidable. It just requires knowing it exists. Now you know.

Add df -i to your monitoring. I am not going to say it a third time.