The conntrack table was full

The failures were intermittent. Not random — intermittent. There is a difference, and it matters for how you debug them, and I did not appreciate the difference soon enough.

Random failures scatter. Intermittent failures have a pattern; you just haven’t found it yet. The connections to the internal services were failing on a schedule that looked random because the schedule was driven by traffic load, and traffic load at 2pm on a Wednesday doesn’t look like a schedule. But it was. The table was filling up on a schedule, and dropping packets on a schedule, and the failures were entirely deterministic once you understood what was driving them.

It took me a day and a half to get there.

What the problem looked like

We ran a cluster of internal services behind a firewall box: a plain Linux host doing NAT and filtering with iptables. Standard setup. Clients in one subnet, services in another, the firewall between them doing masquerade on the internal side. Nothing exotic.

The failures started showing up in the application logs: Connection timed out and Connection refused errors, scattered through the day, no obvious correlation. Not all connections, not to any specific service, not from any specific client. Just occasional failures, getting worse over two or three weeks, until the on-call rotation started getting pages during peak traffic windows.

The obvious stuff was clean. I checked it in order, because you check the obvious stuff first even when you’re pretty sure it’s wrong:

• NIC errors: zero. No input drops, no output drops, no CRC errors, no FIFO overruns on either interface.
• CPU: fine. The firewall box was sitting at 8% CPU during peak, which for a box that is routing and NATing traffic is normal.
• Memory: fine. 16GB RAM, using 4GB.
• iptables rules: correct. I read through every rule in every chain, confirmed the ACCEPT rules matched what we expected, confirmed the MASQUERADE target was in place, confirmed the FORWARD chain had the right policies.
• Routing table: correct. Default routes, static routes for the internal subnets, nothing missing.
• Packet loss on the wire: none. mtr from client to service showed clean paths, 0% loss, consistent RTT.

mtr showed no loss. iptables -L -n -v showed the counters incrementing correctly on the accept rules. The connections were getting through the rules. And then some of them weren’t arriving.

What dmesg was saying

I should have checked this first. I didn’t, because the firewall was dropping packets before I connected the symptoms to the kernel.

dmesg | grep -i conntrack

[1234567.123456] nf_conntrack: nf_conntrack: table full, dropping packet
[1234567.234567] nf_conntrack: nf_conntrack: table full, dropping packet
[1234567.456789] nf_conntrack: nf_conntrack: table full, dropping packet

Hundreds of them. Timestamped through the previous twelve hours, clustered during the exact windows when we were getting application-level failures. The kernel had been telling us the problem in plain English the whole time, in the one log we hadn’t looked at.

The conntrack table was full.

What the conntrack table is

If you’re running iptables with stateful rules — which you are, if you’re using ESTABLISHED,RELATED matches or MASQUERADE — then the kernel is maintaining a connection tracking table. Every connection that flows through the firewall gets an entry: source IP, source port, destination IP, destination port, protocol, state. The kernel uses this table to match return traffic back to its originating connection, to enforce stateful rules, to do NAT mapping.

The table has a maximum size. It is configured at module load time. The default is calculated based on available RAM, but the formula hasn’t been aggressive: on a box with 16GB of RAM you might get a default max of 131072 entries. Call it 128k connections. That sounds like a lot until you have a busy service mesh with thousands of short-lived HTTP connections per second, each connection getting its own entry, entries not expiring instantly when the connection closes.

Check the current state:

# Current entry count
cat /proc/sys/net/netfilter/nf_conntrack_count

# Maximum
cat /proc/sys/net/netfilter/nf_conntrack_max

# Or both at once with labels
sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max

On our box:

net.netfilter.nf_conntrack_count = 131068
net.netfilter.nf_conntrack_max = 131072

Four slots left. We were at 99.997% capacity. New connections were being dropped because there was nowhere to put their tracking entries. The iptables rules accepted them. The kernel dropped them anyway, because it couldn’t track them. Accepted at the rule level and dropped at the table level, which is why iptables -L -v showed counts incrementing on the accept rules while clients were getting timeouts.

Why it filled up

The table fills up for two reasons: too many connections, and entries not expiring fast enough. We had both.

The service mesh had grown. Over the past six months, the number of internal services making connections through the firewall had doubled. But connection count per service hadn’t gone up much; the failure mode crept in as aggregate load crossed the threshold.

The entry expiration is the worse part. The kernel keeps conntrack entries alive for varying lengths of time depending on connection state:

sysctl -a | grep conntrack_timeout

net.netfilter.nf_conntrack_tcp_timeout_established = 432000
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_close = 10
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 120
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 60

That first number: nf_conntrack_tcp_timeout_established = 432000. That’s five days. An established TCP connection that goes idle will keep its conntrack entry for five days by default. In a normal deployment with well-behaved applications that close connections promptly, this is fine. In a deployment where a few hundred services maintain long-lived keep-alive connections and many of those connections are idle, this is how you fill up a 128k-entry table with entries for connections that technically exist but aren’t doing anything.

We had a lot of those. Internal clients were holding established connections to services they only called occasionally, connections that were valid, tracked, eating conntrack table space indefinitely.

The fix

Two parts: increase the table size, and shorten the timeout for idle established connections.

# Increase the max. Rule of thumb: 1024 entries per MB of RAM is conservative.
# 16GB box: 16384 MB * 1024 = 16777216, which is aggressive.
# Start with something sane and adjust.
sysctl -w net.netfilter.nf_conntrack_max=524288

# The hash table size should be 1/4 of the max for reasonable bucket distribution.
# This requires the module to be reloaded or set via module parameter at load time.
# At runtime, you can't change hashsize without reloading nf_conntrack.
# Check the current value:
cat /sys/module/nf_conntrack/parameters/hashsize

# Shorten the established timeout. Five days is insane for an internal NAT context.
# Thirty minutes is aggressive; six hours is reasonable.
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=21600

Make it persist:

cat >> /etc/sysctl.d/99-conntrack.conf << 'EOF'
net.netfilter.nf_conntrack_max = 524288
net.netfilter.nf_conntrack_tcp_timeout_established = 21600
EOF

For hashsize, you set it as a module parameter. On a system using the default kernel modules, add it to the module options:

echo 'options nf_conntrack hashsize=131072' > /etc/modprobe.d/nf_conntrack.conf

That sets the hash table to 131072 buckets, which is 1/4 of the 524288 max: each bucket will average four entries before chaining. The kernel doesn’t require hashsize to be 1/4 of max, but it’s the right ratio — too small and you get long chains with slow lookups; too large and you waste memory on sparse buckets.

After the sysctl changes (no reload needed for max and timeout):

watch -n 1 'sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max'

The count started dropping as old established entries aged out under the shorter timeout. Within an hour it was at 60k. The drops stopped. The application failures stopped.

What you should monitor

The conntrack table is exactly the kind of thing that sits invisible until it’s 100% full and dropping packets. You want a graph of nf_conntrack_count versus nf_conntrack_max. Most monitoring systems don’t collect this by default.

In Prometheus with the node_exporter, it’s:

node_nf_conntrack_entries / node_nf_conntrack_entries_limit

Set an alert at 70%. At 70% you have time to investigate and tune before anything starts dropping. At 99.997% you are already on fire and didn’t know it.

If you’re doing ad-hoc checks, the one-liner:

paste <(cat /proc/sys/net/netfilter/nf_conntrack_count) \
      <(cat /proc/sys/net/netfilter/nf_conntrack_max) | \
awk '{printf "%.1f%% (%d/%d)\n", $1/$2*100, $1, $2}'

If that number is above 50%, you should tune. If it’s above 80%, tune now. If it’s above 90%, you have packets being dropped and you may not know it yet because the drops are intermittent.

The part that bothers me

The kernel was generating log messages. Hundreds of them. nf_conntrack: table full, dropping packet is not a subtle message. It is a complete, accurate, actionable description of the problem. The kernel wrote it to the ring buffer every time it dropped a packet.

We didn’t look at the ring buffer for a day and a half.

I’m not going to pretend this was some exotic failure mode that required deep kernel expertise to diagnose. The diagnosis was dmesg | grep conntrack. That’s it. Everything else was just following the output to the fix. The gap was that nobody’s runbook said “check dmesg” during a network connectivity incident, because dmesg feels like a hardware diagnostic, and network connectivity incidents feel like a network or firewall problem, and the firewall is iptables, and iptables is userspace, and dmesg is kernelspace, and that conceptual boundary means people look in the wrong place.

The conntrack table is not userspace. It’s not part of iptables, exactly — it’s a kernel subsystem that iptables uses. The rules are userspace configuration; the table is a kernel data structure. You can have perfect iptables rules and a full conntrack table, and the kernel will drop packets that your rules explicitly accepted. The rules don’t get the final word. The table does.

Every system running iptables with stateful rules or NAT has a conntrack table. Most of them are running with default sizes that were calculated years ago for traffic patterns that no longer apply. The failures don’t show up until the table fills, the table fills gradually, the fills correlate with traffic peaks and look like random noise, and there’s no alert because nobody set one up, because the table is invisible until it’s full.

The pattern is the same as the TCP window post from a few days ago: a kernel parameter set to a value that was reasonable at some point sits unchanged while the workload grows around it, until the parameter becomes a hard ceiling on system behavior, and then something breaks in a way that looks intermittent and mysterious until you trace it back to the number.

The number here is 131072. Check yours.

sysctl net.netfilter.nf_conntrack_max

If you’re running any stateful iptables rules on a busy host, and you’ve never tuned that number, the default is probably too low for your current workload. You won’t know it’s too low until the table fills. You won’t know the table filled until you check dmesg.

Check dmesg.