The listen queue had been five since 2014

02:47. Pager fires on elevated error rate from the API tier. Not catastrophic: 14% connection failures on one internal service endpoint. The service has retry logic with exponential backoff and the overall error rate stays below the client-visible threshold on most requests. The pager fired because the retry rate crossed a separate alert threshold that someone had added two years ago and never tuned.

I was the second engineer on the call. The first had been on for fifteen minutes and had already ruled out the obvious list: CPU, memory, disk, network throughput, upstream service health. Everything nominal. The error rate was real. The cause was not in any of the dashboards.

The errors themselves were connection refused or connection reset, depending on the client: TCP connections being dropped before the service could accept them.

I ran the standard connection diagnostics.

ss -tlnp:

State   Recv-Q  Send-Q  Local Address:Port  Process
LISTEN  5       5       0.0.0.0:9200        users:("pipeline",pid=7841,fd=4)

Send-Q was 5. Recv-Q was 5. The accept queue was full.

netstat -s | grep -i overflow:

    1847291 times the listen queue of a socket overflowed

1.8 million connection drops. The service had been in production for nine years. The counter had been incrementing since 2016.

What the accept queue is

When a TCP client initiates a connection, the three-way handshake completes inside the kernel. After the final ACK, the kernel places the fully established connection into the accept queue: a buffer of completed connections waiting for the application to call accept(). The application pulls connections off this queue one at a time and handles them.

The accept queue has a maximum size. That maximum is the backlog argument passed to listen(). When the queue is full and a new connection completes its handshake, the kernel drops it. The client either retries the handshake or gets a timeout. The application never sees the connection. Nothing in application logs records it.

The backlog is also capped by net.core.somaxconn. The effective backlog is min(listen_backlog, somaxconn). On most systems somaxconn defaults to 128. The application here was calling listen(sock, 5).

Five. The accept queue held five connections.

Where five came from

git log --all --follow -p -- src/net/server.c | grep -B2 -A5 'listen('

Initial commit. 2014. The comment above the call read /* TODO: make this configurable */. There is a long tradition of TODO comments in initial commits that are never revisited because the code works well enough in development, and in development, nobody is firing 400 connections per second at a single service with a 5-slot accept queue.

The service hit production in 2016. For a year, engineers noticed intermittent connection failures under load. The diagnosis was “transient network instability.” In 2017 an engineer added retry logic to the calling service, the connection failures became invisible to clients, and the ticket was closed as resolved. The backlog stayed at 5.

From 2016 to 2026: nine years of retries, 1.8 million dropped connections, zero connection-layer alerts, zero root-cause investigations, zero complaints. The service had a 14% connection drop rate at peak load and everyone had adapted to it without knowing that’s what they were adapting to.

The stats that tell you

netstat -s | grep -i "listen\|overflow\|drop"

    1847291 times the listen queue of a socket overflowed

That number is a monotonic counter in /proc/net/netstat under the key TcpExtListenOverflows. It increments every time a TCP connection is dropped because the accept queue was full. It requires no configuration to read. netstat -s is the human-readable wrapper. The counter starts at zero on boot and accumulates.

The service had been incrementing this counter since 2016. The counter was visible to anyone who ran netstat -s on that host. Nobody ran netstat -s on that host during normal operations because nothing was obviously broken.

ss -tlnp | grep ':9200'

In ss -tlnp output for a LISTEN socket, Recv-Q is the current number of connections sitting in the accept queue waiting to be accept()ed. Send-Q is the effective backlog: the maximum before the kernel starts dropping. When Recv-Q reaches Send-Q, the queue is full and the next completed handshake gets dropped.

The Send-Q had been 5 on that line since 2016.

The fix

Three changes.

First, fix the listen() call:

if (listen(server_fd, 1024) < 0) {
    perror("listen");
    exit(EXIT_FAILURE);
}

1024 is a reasonable default for a service handling short-lived connections under real load. For services where accept() is slow or where bursts are large, go higher. The cost is a small amount of kernel memory per queued connection.

Second, net.core.somaxconn:

sysctl net.core.somaxconn
# net.core.somaxconn = 128

An application backlog of 1024 is silently capped to 128 by a somaxconn of 128. The application thinks it set 1024. The kernel enforces 128. ss -tlnp shows the actual effective value, not the value the application requested.

sysctl -w net.core.somaxconn=1024
echo "net.core.somaxconn = 1024" >> /etc/sysctl.d/50-network.conf
sysctl -p /etc/sysctl.d/50-network.conf

Third, for services under heavy SYN load, net.ipv4.tcp_max_syn_backlog. This is the incomplete-connection queue: connections that have received a SYN but not yet completed the handshake. It is a separate queue with a separate limit, defaulting to 128 or 256 on most systems.

sysctl -w net.ipv4.tcp_max_syn_backlog=2048

A TCP connection has to clear both queues on its way from SYN to accept(). If either queue is full, the connection fails. The two parameters are unrelated in the source but both show up in the same incident.

After the changes: ss -tlnp | grep ':9200' showed Send-Q 1024. netstat -s | grep overflow stopped incrementing. The retry rate on the calling service dropped to zero within two minutes.

The counter read 1,847,291. Then 1,847,291 for the next hour. Then 1,847,292. Someone had deployed an unrelated service on the same host. I checked its listen() call.

What to check before a TCP service handles load

1. Read the listen() call. If the backlog is a literal integer from a tutorial (5, 10, 50), it probably never changed from initial implementation. Make it configurable and set it appropriately for the connection rate and accept() latency of the service.

2. Run sysctl net.core.somaxconn. On most current kernels this defaults to 4096, but on older hosts it may be 128. The effective backlog is the lower of the listen() argument and this value. ss -tlnp shows you the actual effective backlog, not what the application believes it set.

3. Run netstat -s | grep overflow on any production host handling TCP connections. If that counter is nonzero and you do not know why, you have a service with a full accept queue somewhere on that host. Find it with ss -tlnp and look for a Send-Q that seems small relative to the service’s connection rate.

4. Add TcpExtListenOverflows from /proc/net/netstat to whatever you collect from every host. It is a single integer. It should be zero or very close to zero. If it is incrementing, connections are being dropped. There is no benign reason for a busy production service to have a full accept queue.

The pattern holds

netstat -s showed 1.8 million dropped connections. ss -tlnp showed Send-Q of 5. Both had been true for the full nine years the service was in production. The retry logic added in 2017 masked the symptom so thoroughly that no one investigated the cause for another nine years.

The retries were individually correct: the client got its response on the second or third attempt, the user saw acceptable latency, the ticket was closed resolved. What they bought was a permanent 14% overhead on every burst, compounding, for nine years, along with a complete absence of visibility into the actual failure mode. The one-time cost of reading the listen() man page in 2014 would have been cheaper.

The kernel had the right number the whole time. Same as the ephemeral port range with 800 slots left, the number 1024 that nobody questioned, the conntrack table at 24 entries: the resource limit that puts you in an incident is readable in one command, updated in real time, free. The number the kernel keeps for you is always more current than the number in your documentation. Read it before you need it.

TcpExtListenOverflows: 1,847,291. Nine years. One listen() argument.