"Cannot assign requested address" is not a DNS problem. You're out of ports.

Under load the application starts throwing EADDRNOTAVAIL — “Cannot assign requested address.” People read “address” and go looking at DNS or the interface config. Neither is involved.

The kernel is telling you it has no source port left to give you.

The arithmetic

Every outbound TCP connection needs a unique 4-tuple: source IP, source port, dest IP, dest port. Connecting to one backend from one interface fixes three of those, so your only degree of freedom is the source port — and the ephemeral range on Linux defaults to:

sysctl net.ipv4.ip_local_port_range
net.ipv4.ip_local_port_range = 32768	60999

28,231 ports. That’s your hard ceiling on concurrent connections to a single destination IP:port from a single source IP.

Worse: when your side closes the connection, the tuple enters TIME_WAIT and stays there for 2 × MSL — 60 seconds on Linux, not tunable via sysctl. So the real limit isn’t concurrency, it’s rate: 28,231 ports ÷ 60 seconds ≈ 470 new connections per second to one backend, sustained, before you start failing.

A service doing 800 req/s to a single upstream with connection reuse disabled will fall over reliably and look, from the application’s perspective, like the upstream went away.

Confirm it in one command

ss -tan state time-wait | wc -l
sysctl net.ipv4.ip_local_port_range

If that count is within spitting distance of the range size, that’s your bug. For the per-destination view — because the limit is per-tuple, not global:

ss -tan | awk '{print $5}' | sort | uniq -c | sort -rn | head

One destination with 25,000 connections against it is the whole story.

And check the counters, which name the problem outright:

nstat -az | grep -iE 'TcpExtTW|PortFail|TcpExtTCPTimeWaitOverflow'

The fixes, worst to best

3. Widen the range

sysctl -w net.ipv4.ip_local_port_range="10240 65535"

Buys you about 2× and takes thirty seconds. Do it — it’s free and there’s no reason for the default’s conservatism on a modern box. But you have doubled a limit you are hitting by design, so you’ve bought a few months, not a fix. Keep it above 1024 and out of the way of anything you bind explicitly.

2. Recycle TIME_WAIT faster

Here be dragons, and most of the advice online is actively wrong.

net.ipv4.tcp_tw_reuse=1 lets the kernel reuse a TIME_WAIT socket for a new outbound connection when timestamps show it’s safe. This is the safe one. Turn it on.

net.ipv4.tcp_tw_recycle is the one every 2013 blog post recommends. It was removed from Linux in 4.12 because it dropped connections from any client behind NAT — it keyed timestamp state per source IP, and NAT means many hosts share one. If a runbook in your repo still sets it, that runbook is old enough to drive.

Do not “fix” TIME_WAIT by shortening it below the standard on the assumption nothing bad will happen. TIME_WAIT exists so a delayed duplicate packet from a dead connection can’t be accepted into a new one that reused the tuple. That failure mode is data corruption at the application layer and it is extremely fun to debug.

1. Stop opening so many connections

This is the actual fix and the other two are stalling.

Every outbound HTTP client you own has a connection pool, and the default is usually wrong:

• Go — http.Transport{MaxIdleConnsPerHost: 100}. The default is 2, which means a high-throughput service closes and reopens constantly. This one specific default has caused more port exhaustion than anything else in the ecosystem.
• Python requests — HTTPAdapter(pool_maxsize=100) mounted on the session, and use a Session at all. A bare requests.get() opens and closes a connection every call.
• JVM — connection pool config on your HTTP client, and check that keep-alive is actually negotiated end to end.
• curl in a shell loop — that’s one connection per iteration by construction. --keepalive won’t help across processes. Batch it or use a tool that persists.

And check the other end: if your upstream sends Connection: close, no client-side pooling helps. Load balancers get configured that way by accident and it’s invisible until you look at the response headers.

Fix the pool and the graph goes flat. I have watched a 28,000-connection TIME_WAIT pile become 40 persistent connections with a one-line config change.

The variant that looks identical

Same symptom, different table: conntrack exhaustion on a box doing NAT or running a stateful firewall.

sysctl net.netfilter.nf_conntrack_count net.netfilter.nf_conntrack_max
dmesg | grep -i 'nf_conntrack: table full'

When that table fills, packets get dropped — not rejected, dropped — so the client sees a timeout rather than an error, and the connection failures appear at a completely different layer than the box that’s actually out of room. Raise nf_conntrack_max, and if the traffic doesn’t need connection tracking, exempt it with NOTRACK in the raw table rather than sizing the table for traffic you were never going to inspect.

The tell

The reason this one wastes so much time is that the error message points at the network and the cause is in the application’s connection handling. ss -tan state time-wait | wc -l settles it in one command. If that number is near 28,000, stop reading the DNS logs.