The TCP window that ate a gigabit
Published by RodHat
The circuit came up clean. The network team handed it over with a printout: 1Gbps dedicated, zero packet loss, 18ms RTT between the two facilities. They had tested it with their own gear and shown us the graphs. We scheduled the maintenance window, did the cutover, and ran the first real workload across it: a weekly rsync of the snapshot archive. 180Mbps. We’d been running 100Mbps over the old shared tunnel. 180 felt like an improvement.
It was an improvement. It was also a fifth of what we had paid for.
What the network team said
“The link is fine.” They were right. I spent two days trying to prove they were wrong and came up empty. The hardware was clean: no CRC errors on the interface, no input drops, no output drops, no FIFO errors. The routing was fine: 18ms RTT, symmetric in both directions, no asymmetric routing, no ECMP hash collision sending half the flows through a broken path. I ran mtr for six hours and watched it. It was boring. It was correct.
I did the obvious next thing: ran iperf3 from the server side to a test box in the other facility.
# server side
iperf3 -s
# client side
iperf3 -c 10.2.0.10 -t 30[ ID] Interval Transfer Bitrate
[ 5] 0.00-30.00 sec 648 MBytes 181 Mbits/sec sender
[ 5] 0.00-30.00 sec 646 MBytes 181 Mbits/sec receiver181Mbps. Thirty seconds of data, as consistent as a metronome. Not 180, not 200, not varying at all. A flat ceiling at 181Mbps, every run, regardless of which direction I ran the test, regardless of whether I used TCP or UDP.
Wait: UDP was different.
iperf3 -c 10.2.0.10 -u -b 1G -t 30[ ID] Interval Transfer Bitrate Jitter Lost/Total
[ 5] 0.00-30.00 sec 3.32 GBytes 952 Mbits/sec 0.035 ms 1248/2442523 (0.051%)952Mbps with UDP. 181Mbps with TCP. Same wire. Same moment. The link was not the problem.
What the math was saying
I should have remembered this earlier. I knew it, in the same way you know things you learned from a textbook in 1998 and haven’t had to apply since. The bandwidth-delay product is not a suggestion.
The bandwidth-delay product is the amount of data that can be “in flight” on a network path at any given time. It is the product of the link’s bandwidth and the round-trip time. For our link:
bandwidth = 1Gbps = 125 MB/s
RTT = 18ms = 0.018s
BDP = 125 MB/s * 0.018s = 2.25 MBTo fill a pipe to its rated speed, the sender needs to keep 2.25MB of data in flight at all times. TCP does this via the receive window: the receiver tells the sender how much buffer space it has available, and the sender cannot have more than that much unacknowledged data outstanding at any time. This is the flow control mechanism. It also, when the receive window is smaller than the BDP, becomes the throughput limit.
The default Linux TCP receive buffer maximum: net.core.rmem_max. Check it on almost any Linux system that nobody has tuned:
sysctl net.core.rmem_maxnet.core.rmem_max = 212992208 kilobytes. That number has been the default since before most of the machines in this story were installed. At 18ms RTT, it caps throughput at:
208 KB / 0.018s = 11.6 MB/s = 92 MbpsWait, that implies a lower ceiling than what we were seeing. The actual measured ceiling of 181Mbps suggests the effective window was closer to 400KB in practice, which is plausible: Linux auto-tunes within the bounds set by rmem_max, and the tcp_rmem max parameter on these boxes was set to 4MB, which the kernel can use dynamically up to… except net.core.rmem_max is the hard cap. On these specific machines the value was different from the default; someone had set net.core.rmem_max = 425984 at some point, probably from a Stack Overflow answer someone copy-pasted during a previous performance investigation that never got followed up on.
425984 bytes is 416KB. At 18ms RTT: 416KB / 0.018s = 23.1MB/s = 185Mbps. There is the ceiling. We were seeing 181Mbps because of protocol overhead. The math worked out exactly, and the value had been sitting in /etc/sysctl.d/99-network.conf doing its job for years.
The UDP test wasn’t subject to the receive window constraint, which is why it went straight to wire speed. UDP doesn’t have flow control. It just sends. If the receiver can’t keep up, the packets drop, and the caller gets a jitter number and a loss percentage. The link was fine. TCP’s own flow control was the entire problem.
The fix
Three lines:
sysctl -w net.core.rmem_max=16777216
sysctl -w net.ipv4.tcp_rmem="4096 87380 16777216"
sysctl -w net.core.wmem_max=16777216That sets the socket receive buffer maximum to 16MB per socket, which is more than enough headroom for any reasonable RTT across our infrastructure. Then:
iperf3 -c 10.2.0.10 -t 30[ ID] Interval Transfer Bitrate
[ 5] 0.00-30.00 sec 2.98 GBytes 854 Mbits/sec sender
[ 5] 0.00-30.00 sec 2.97 GBytes 852 Mbits/sec receiver854Mbps. The link the network team had sold us. Consistent across multiple runs. The rsync job that had been taking six hours took fifty minutes.
Persist it:
cat >> /etc/sysctl.d/99-network.conf << 'EOF'
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
EOFDone. Four parameters. Years of paying for a gigabit and using 18% of it.
The part that bothers me
We had been running the snapshot rsync over that facility pair for two years before the dedicated circuit. The VPN tunnel ran at 100Mbps because the shared link was 100Mbps. When we got the upgrade, we celebrated the 180Mbps number because it was 80% better than before. Nobody asked why it wasn’t 800%.
The answer is that 180Mbps felt like a success after 100Mbps. We had a reference point that was wrong, and the wrong reference point made the broken result look good. This is how a lot of performance bugs survive: not because they’re hard to find, but because nobody looks until something forces a comparison against the actual spec.
The thing that forced the comparison was a failed SLA conversation. The business was asking why a nightly cross-facility backup was taking six hours when we had a gigabit link. I had to go find the answer, which meant actually running iperf3 for the first time on that path, which meant seeing 181Mbps and knowing immediately that something was wrong, which eventually led to sysctl net.core.rmem_max and a very quiet fix.
The storage investigation that looks at one metric while the real problem is somewhere else is a recurring theme. The pattern: a number looks acceptable in isolation, the real constraint is invisible until you check against spec.
What ss was showing all along
The thing is, this was visible without iperf3. You can see the TCP receive window advertisement directly in socket state. On the receiving end, while rsync was running:
ss -tin dst 10.2.0.10State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 10.1.0.5:873 10.2.0.10:54821
ts sack wscale:7,7 rto:220 rtt:18.4/2.1 ato:40 mss:1448 pmtu:1500
rcvwnd:408960 sndwnd:408960 rcvspace:425984 sndspace:425984rcvwnd:408960 is the current receive window: 400KB. That number, divided by the RTT, is your throughput ceiling. If you know where to look in ss output, the constraint is sitting right there. I didn’t know to look there; I went to iperf3 first because UDP vs TCP comparison is the fastest way to rule out the physical link.
The rcvspace:425984 is the socket buffer allocation, which matches net.core.rmem_max exactly. The kernel had hit the configured ceiling and stopped there. The receive window advertisement was accurate: the kernel had exactly 400KB of buffer space available, the receiver advertised exactly 400KB, and the sender dutifully capped its outstanding data at 400KB.
The protocol was working perfectly. The parameters were wrong.
Calculating it yourself
Before touching a long-distance link, do the BDP calculation. It takes thirty seconds and will tell you whether the default parameters are adequate.
# RTT in seconds (measure with ping)
rtt=0.018
# Link speed in bytes per second
bw=$((1024 * 1024 * 1024 / 8)) # 1Gbps in bytes
# BDP in bytes
bdp=$(echo "$bw * $rtt" | bc)
echo "BDP: ${bdp} bytes ($(echo "$bdp / 1024 / 1024" | bc)MB)"BDP: 2281701 bytes (2MB)If net.core.rmem_max is less than that number, you are leaving throughput on the floor. On any link with RTT over about 3ms and bandwidth over 100Mbps, the defaults are almost certainly wrong. On cross-datacenter links with 15-50ms RTT and gigabit or better capacity, they are definitely wrong.
The fix is not risky. Larger socket buffers cost memory per connection: 16MB of rmem_max means the kernel can allocate up to 16MB per receiving socket, but it only allocates what it needs up to that cap, and it won’t allocate the full 16MB for connections that don’t need it. On a box with 32GB of RAM handling a few dozen active connections, this is not a tuning tradeoff, it is just correct. The default exists because it was chosen for a world where memory was measured in megabytes and WAN links were rare. That world is gone. The default stayed.
tcpdump on the receiving interface during a slow transfer will show you window advertisements in the TCP headers if you want to confirm this directly before changing anything. Look for the win field in the output; if it’s consistently small and not varying much, the receive side is the bottleneck.
The thing about defaults
The number 212992 was not chosen to be a performance ceiling. It was chosen in a time when most hosts were not doing bulk data transfer over high-latency paths, and when RAM was expensive enough that a 200KB per-socket cap was a reasonable default to avoid buffer bloat on machines with 64MB of total RAM.
Linux auto-tuning has improved this substantially since then: with tcp_moderate_rcvbuf enabled (the default), the kernel dynamically adjusts socket buffers up to the configured maximum based on measured bandwidth and RTT. The problem is “up to the configured maximum.” If the maximum is 416KB, the autotuning ceiling is 416KB, and no amount of kernel cleverness will help.
The BDP is physics. The window size is a configuration parameter. If the parameter is smaller than the physics demands, you get exactly the throughput the parameter allows and not one bit more, and the link you paid for sits mostly idle, and the backup job takes six hours, and nobody notices for two years because 180Mbps was better than 100Mbps and better was good enough.
It wasn’t good enough. Run the calculation. Fix the parameter. The math does not negotiate.