The config file changed and nothing reloaded

The deployment went fine. The config change was applied. The service didn’t reload.

Not a crash: the process was running, serving traffic, responding to health checks. It was just serving the old config. The new rate limits were supposed to take effect at 14:00. At 14:47 someone noticed they hadn’t. The watcher daemon, whose entire job was to detect config file changes and signal the service to reload, had been running for seven months. It just hadn’t been detecting anything for the last two of them.

What the watcher was supposed to do

The setup: a small daemon, written by a former colleague who had since left, using inotify to watch a directory of per-service configuration files. When any file in the directory changed, the daemon sent SIGHUP to the appropriate service to trigger a config reload. Standard Unix pattern. Editor writes config, daemon sees change, service gets signal, service reloads. No polling, no restart required, change applied inside a second.

It worked fine for five months. Then it silently stopped.

No crash. No error in the application logs. The daemon was alive; ps confirmed it, systemctl status showed it green, the heartbeat endpoint it exposed returned 200. Configuration changes were being made and committed and deployed. Services were not reloading. The ops team had been manually restarting services after config changes for two months without, apparently, asking why the automatic reload had stopped working.

That last part is a different problem. This story is about the inotify part.

The investigation

I started where you start: strace.

strace -e trace=read,inotify_add_watch -p $(pgrep config-watcher)

Then, in another terminal, I wrote a whitespace change to a watched config file and saved it.

Nothing. No inotify_add_watch call. No read() returning event data. The process was completely silent from a syscall perspective. It was alive, using CPU for nothing, watching nothing.

The daemon had a health endpoint that returned a JSON blob with watches_active in it. I curled it.

{"status":"ok","watches_active":8192,"watches_failed":3847}

watches_failed: 3847.

That field was there. In the health endpoint. Nobody had looked at it.

The limit

inotify is the Linux kernel mechanism for watching filesystem events. A process calls inotify_create() to get a file descriptor, then inotify_add_watch() to register a path. When something happens to that path, the kernel queues an event readable from the inotify file descriptor. No polling, immediate notification.

The kernel limits the number of watches a user can hold across all inotify instances. The limit lives here:

cat /proc/sys/fs/inotify/max_user_watches
8192

8192. That is the default. It was chosen in approximately 2005, for desktop Linux, for use cases like a file manager noticing when you add files to your home directory.

The config-watcher daemon was watching a service config directory that had grown to 8192 entries. When entry number 8193 got added, inotify_add_watch() returned -1 with errno set to ENOSPC. The daemon logged this at DEBUG level, which was turned off in production, and continued running, healthy, watching nothing new. Every config change to any directory added after that point was invisible to it.

strace -e trace=inotify_add_watch config-watcher 2>&1 | grep ENOSPC
inotify_add_watch(3, "/etc/services/new-ratelimiter", IN_MODIFY|IN_CREATE) = -1 ENOSPC (No space left on device)
inotify_add_watch(3, "/etc/services/auth-v3", IN_MODIFY|IN_CREATE) = -1 ENOSPC (No space left on device)
[...]

ENOSPC on a disk with 80 gigabytes free. The kernel returned “no space left” because the inotify watch table was full, not the disk. Same ambiguity as a process seeing ENOSPC when it’s out of inodes rather than blocks. Same error code, completely different problem, no way to distinguish it from the application side without reading the errno and knowing what ENOSPC means for this particular syscall.

The watches_failed counter sitting in plain sight

The health endpoint had been reporting watches_failed: 3847 for two months. Three thousand eight hundred forty-seven times the daemon tried to add a watch and got ENOSPC. Nobody had added that field to the monitoring alert configuration. Nobody had looked at the health endpoint except to verify status: ok.

A counter named watches_failed with a value of 3847 is not a healthy service with one field in the red. It is a service that has been silently failing 3847 times. status: ok was technically accurate: the daemon was alive and its HTTP server was responding. It was providing zero information about whether the daemon was doing its actual job.

This is the same failure pattern as a service reporting connections_established: 100000 and connections_dropped: 99847 and considering itself healthy because the uptime is good.

Counting watches manually

You do not need the application to tell you how many watches it holds. The kernel tracks this in /proc.

PID=$(pgrep config-watcher)
for fd in $(ls /proc/$PID/fd); do
    target=$(readlink /proc/$PID/fd/$fd 2>/dev/null)
    if echo "$target" | grep -q "^anon_inode:inotify"; then
        wc -l /proc/$PID/fdinfo/$fd
    fi
done

The fdinfo entry for an inotify file descriptor includes one line per registered watch. Count the lines, that is the watch count.

There are three knobs in /proc/sys/fs/inotify:

cat /proc/sys/fs/inotify/max_queued_events   # event queue size per fd (default: 16384)
cat /proc/sys/fs/inotify/max_user_instances  # inotify fds per user (default: 128)
cat /proc/sys/fs/inotify/max_user_watches    # watches per user (default: 8192)

Check all three when something breaks. The queued events limit causes events to be dropped under high-churn conditions even when the watch count is fine. The user instances limit will hit you if a process leaks inotify file descriptors without closing them. The watch limit is the one that most people hit first because 8192 is small.

The fix

sysctl -w fs.inotify.max_user_watches=524288

Persist it:

echo 'fs.inotify.max_user_watches=524288' >> /etc/sysctl.d/99-inotify.conf
sysctl -p /etc/sysctl.d/99-inotify.conf

524288 is 2^19. It is not a number chosen for a specific reason other than being much larger than 8192 and small enough not to matter on a machine with 32 gigabytes of RAM. Each watch entry costs roughly 1 kilobyte of kernel memory. 524288 watches is roughly 512 megabytes if fully populated. Use it all and you will notice. On a machine with less than 4 gigabytes of RAM, pick a smaller ceiling and ask yourself why you have that many watched paths.

After raising the limit, restart the daemon. The inotify_add_watch() calls that had been returning ENOSPC will now succeed. The backlog of unwatched paths will not be automatically repopulated: the daemon needs to re-register all the paths it missed. Most daemons that handle this correctly will re-enumerate the watch directory on startup. This one did. If yours does not, restart it and verify the watch count came up to the expected value.

What the default number means

8192 is an early 2000s number. 1024 was a different era’s number in a different place. The listen backlog that had been five since 2014 was a provisioning assumption that never got revisited. These defaults accumulate across production systems and most of them sit quietly until the workload grows past where they made sense.

The 8192 watch limit made sense for a workstation running a file manager. A deployment system watching config directories for hundreds of services is a different workload. The mismatch was always there; it just took five months to become a 14:47 page.

The documentation for inotify_add_watch(2) lists ENOSPC as a return value. The man page says, plainly: “The user limit on the total number of inotify watches was reached or the kernel failed to allocate a needed resource.” The limit and its sysctl knob are documented in the kernel source and in the Linux man pages going back to 2.6.13. Nothing about this failure was undocumented or obscure. The number just never got checked because nothing failed loudly enough to make anyone look.

The two fixes

The sysctl is the fast fix. The real fix is the monitoring.

A daemon with a watches_failed counter in its health endpoint should alert on that counter being nonzero. “Should this be nonzero” has a clear answer when the field is named watches_failed. That is not a hard decision to make. The hard part is making it before the incident, not after.

If you have any daemon that watches files: look at what it reports in its health or metrics endpoint, find anything named failed or error or dropped, and check whether those fields are in your alert configuration. If watches_failed is currently nonzero on your system, now you know what to do about it.

If you do not know your max_user_watches value, now is a fine time to check.