$RodHat_
Rod's Tales

The service was healthy and accepting no new connections

Published by

The service was healthy and accepting no new connections
Photo: AI-generated — no human photographer / RodHat AI Cover

23:14. The alerting channel fired: elevated connection refused rate from the API tier. Seven hundred connection refusals in four minutes. The services team’s first instinct was firewall. I’d seen this enough times to know better.

curl -s https://api.internal/health
{"status":"ok","uptime":8312847,"requests_served":14823001}

The health check passed. Requests were being served. The process was not down. The process was not restarting. The port was open:

ss -tlnp | grep 8443
LISTEN  0  1024  0.0.0.0:8443  0.0.0.0:*  users:(("api-service",pid=9847,fd=7))

Listening, backlog of 1024. No firewall change in the past six hours. The host was reachable. Existing sessions were active. New connections from clients were getting connection refused from the service itself, not from anything in front of it.

What accept() was returning

If a process is listening on a port but refuses new connections, the place to look is at the syscall level.

strace -e trace=accept,accept4 -p 9847 2>&1 | head -20
accept4(7, {sa_family=AF_INET, sin_port=htons(58432), sin_addr=inet_addr("10.4.2.71")}, [16], SOCK_CLOEXEC) = -1 EMFILE (Too many open files)
accept4(7, {sa_family=AF_INET, sin_port=htons(58433), sin_addr=inet_addr("10.4.2.88")}, [16], SOCK_CLOEXEC) = -1 EMFILE (Too many open files)
accept4(7, {sa_family=AF_INET, sin_port=htons(58434), sin_addr=inet_addr("10.4.2.71")}, [16], SOCK_CLOEXEC) = -1 EMFILE (Too many open files)

EMFILE. Not a firewall. Not a network partition. The process was calling accept4() on the listening socket, the kernel was handing it a new connection, and the process was getting EMFILE because accepting a connection requires allocating a file descriptor and there were none left.

The TCP three-way handshake had already completed by the time accept4() ran. The client saw the SYN-ACK. The connection was established from the client’s perspective. Then the server tried to allocate a file descriptor for it, failed, and closed the connection. From the client side, that shows up as connection refused. The service accepted the TCP handshake and then immediately closed the connection. Seven hundred clients, seven hundred times.

The limit

cat /proc/9847/limits
Limit                     Soft Limit           Hard Limit           Units
Max open files            1024                 1048576              files

Soft limit: 1024. Hard limit: 1,048,576. The soft limit is what the process actually enforces. The hard limit is the ceiling it’s allowed to raise itself to. The soft limit was 1024.

ls /proc/9847/fd | wc -l
1023

One thousand and twenty-three file descriptors open. The limit was one thousand and twenty-four. One file descriptor to spare, and it needed that descriptor for every incoming connection.

The soft limit of 1024 goes back to early BSD Unix. It made sense in 1983 when a workstation had 4 megabytes of RAM and running a few hundred simultaneous connections was not a thing anyone was doing. The kernel default, the ulimit -n default in most shells, the default in a lot of init systems: 1024. In 2026, on a machine with 32 gigabytes of RAM running a service that handles tens of thousands of concurrent connections, 1024 is a trap left over from when computers were furniture.

The process’s startup script set ulimit -n 1024 explicitly, because whoever wrote it had copy-pasted it from a reference script they found somewhere in the internal wiki, and the reference script had been written in 2017 by someone who had copy-pasted it from a Stack Overflow answer, and the Stack Overflow answer was from 2012, and 1024 was apparently just the example that was there.

Why it had 1023 file descriptors open

That number was the interesting part. The service had no business holding 1023 file descriptors at steady state. Under load it had maybe 300 active connections. Each connection is one file descriptor. Add a few for the listening socket, stdout, stderr, some log files. You should be well under 400, not at 1023.

ls -l /proc/9847/fd | awk '{print $NF}' | sort | uniq -c | sort -rn | head -20
     618 /var/log/api-service/access.log
     300 socket:[...]
       3 /var/log/api-service/error.log
       1 /dev/null
       1 anon_inode:[eventpoll]
       1 pipe:[...]
       1 socket:[...] (listening)

618 open file descriptors pointing at the access log.

The logging library had a rotation handler. When logrotate ran at midnight and renamed access.log to access.log.1, the library was supposed to detect the rename, close the old file descriptor, and open a new one for the new access.log. The handler had a bug: it opened the new file, started writing to it, and never closed the old descriptor. Each rotation added one more dangling file descriptor pointing at the now-renamed old log file. Logrotate ran nightly, so the leak accrued one descriptor per day. The service had been running for 618 days without a restart.

One file descriptor per day, for 618 days, eventually gets you to 1023.

The leak was not new. The leak had existed since deployment. Nobody had noticed because the service was well under the 1024 limit for a year and a half, then crossed 900 three months ago and kept climbing, then hit 1023 and started refusing connections. The last twenty-five descriptors had been the ones that mattered. The first 593 had been invisible.

Reading the fd table without strace

You do not need strace to see this. The fd table is in /proc:

ls -l /proc/$PID/fd

Each entry is a symlink to what the process has open. Sockets show up as socket:[inode]. Regular files show up as the path. Pipes show up as pipe:[inode]. If you see a lot of descriptors pointing at the same log file, that is a rotation leak. If you see a lot of sockets in CLOSE_WAIT with no corresponding listener, that is something else.

To get the limit and current usage in one shot:

PID=$(pgrep api-service)
echo "open: $(ls /proc/$PID/fd | wc -l)"
echo "limit: $(cat /proc/$PID/limits | awk '/open files/{print $4}')"

If open is within 10% of limit, you will eventually have this incident.

The two fixes

The fast fix: raise the soft limit. You cannot raise it past the hard limit without root. The hard limit was 1,048,576, so there was room.

prlimit --pid 9847 --nofile=65536:1048576

prlimit can update the limits of a running process without restarting it. The change takes effect immediately. The process started accepting connections within seconds. Seven hundred clients retried; they got through.

Persist the fix so it survives a restart. Where you set this depends on how the service is managed. If it is a supervised process under runit or s6, put it in the run script:

exec chpst -o 65536 ./api-service

chpst -o sets RLIMIT_NOFILE before exec-ing the service. No systemd required.

The real fix: find and close the file descriptor leak. The logging library’s rotation hook needed to close the old file descriptor after opening the new one. One line. That fix went into the next release.

Without fixing the leak, raising the limit to 65536 buys you another 178 years at one descriptor per day. Probably fine. But close the leak anyway, because the descriptor count growing without bound is a resource leak, and resource leaks eventually compound with something else you have not found yet.

What to monitor

The combination you want:

# Current count vs limit, per process
for pid in $(pgrep -d' ' api-service); do
    open=$(ls /proc/$pid/fd 2>/dev/null | wc -l)
    limit=$(awk '/Max open files/{print $4}' /proc/$pid/limits 2>/dev/null)
    echo "$pid: $open / $limit"
done

Alert when the ratio exceeds 80%. That gives you time to investigate before the service starts refusing connections. At 1023/1024 you have no time. At 819/1024 you have a week.

Also: grep your logs for EMFILE and too many open files. Those strings appear in error logs from almost any language runtime when a file descriptor operation fails. If they are present and not in any alert, you are flying blind.

The service had been logging EMFILE: accept failed at WARN level for four minutes before the alerting channel fired on connection refusals. The log was there. The string was there. Nobody had written an alert for it.

The number 1024

The listen backlog that had been five since 2014 was a provisioning default that outlived its usefulness. The inotify watch limit of 8192 was a 2005 desktop-Linux assumption applied to server workloads. The 1024 open-file limit is the same category of thing: a number that was chosen decades ago for hardware and workloads that no longer exist, persisting because nobody changed it, waiting for the day the workload outgrows it.

These numbers do not advertise themselves. They sit in /proc/$PID/limits, in sysctl output, in startup scripts with comments that are two authors removed from whoever made the original decision. They fail quietly until they do not fail quietly, at which point they fail at the worst possible time.

Check your long-running services. ls /proc/$PID/fd | wc -l. Check the limit. Check whether the count is trending up. If it is trending up on a process that has a stable connection count, there is a leak, and it will become your problem at a time of the leak’s choosing rather than yours.

618 days is a long time to let a leak run before it matters. It’s also not that long at all.