Topic · 33 articles
Networking & DNS
"It's the network" is where most debugging stops and most blame starts. It's usually not the network. When it is, the evidence is already sitting in kernel counters, packet captures, and the DNS delegation chain, and none of it needs a vendor login to read.
Start by finding out what's actually listening and what the TCP stack thinks of each connection. Then DNS, packets, and TLS by hand. The Tales are a long list of tables that filled up quietly: conntrack, ephemeral ports, listen backlogs, and one TCP window.
Start here
Read these first, in this order.
- 01
Tip · 2026-07-18
Stop guessing what's listening on that port
netstat -an tells you a socket is open. It does not tell you which process opened it, which user owns it, or whether the accept queue is already overflowing. ss and sockstat do, and they take the same three seconds.
- 02
Tip · 2026-08-19
ss -i shows you what netstat never could: TCP internals live
Everyone knows ss -tulnp. Almost nobody uses ss -i, which surfaces congestion window size, RTT, retransmit counts, and send/receive buffer fill, directly from the kernel, no tool required.
- 03
Tip · 2026-08-10
dig +trace, and how to tell whose DNS is actually lying to you
"It's a DNS problem" is where debugging stops. +trace walks the delegation from the root yourself, +norecurse asks a resolver what it has cached without letting it go fetch, and together they tell you whether the bad answer is the zone, the resolver, or the client.
- 04
Tip · 2026-08-04
Stop waiting beside tcpdump like it owes you an incident
Intermittent network failures do not happen while you are watching. Use tcpdump's rotating capture files, size limits, and post-trigger preservation to keep the packets that existed before the pager fired.
- 05
Tip · 2026-07-29
Debug a TLS handshake by hand with openssl s_client
"Certificate verify failed" is four different bugs wearing one error message. s_client shows you the chain the server actually sent, in order, with the verify result per link, which is how you find the missing intermediate in about twenty seconds.
- 06
Tip · 2026-08-02
"Cannot assign requested address" is not a DNS problem. You're out of ports.
A client making thousands of short-lived outbound connections exhausts the ephemeral range, and every tuple sits in TIME_WAIT for a minute afterwards. Here's how to confirm it in one command and the three fixes, ranked by how much you'll regret them.
The syscalls underneath
Version introduced, sourced from the man page. Full cards in the syscall reference, or check your kernel.
| Interface | Since | What it does |
|---|---|---|
| SO_REUSEPORT | Linux 3.9 | Let multiple sockets bind one address:port; the kernel spreads traffic across them. |
| SCM_RIGHTS | unknown | Pass open file descriptors to another process over a UNIX domain socket. |
More console tips (7)
- SO_REUSEPORT: each worker gets its own accept queue, and then the migration problem hits
- SCM_RIGHTS: send an open file descriptor to another process over a Unix socket
- ip netns is what container networking actually is. No daemon required.
- socat is the Swiss Army knife you keep reaching past
- Two uplinks, one box, and the reply going out the wrong interface
- Let pf keep the ban list so you don't have to
- Your ssh is reconnecting every single time. It doesn't have to.
Rod's tales (9)
- The code ran in three milliseconds and the request took two hundred
- The listen queue had been five since 2014
- The ephemeral port range had eight hundred slots left
- The conntrack table had twenty-four slots left
- The conntrack table was full
- The TCP window that ate a gigabit
- The pool was full of dead connections
- The migration was flawless. The TTL was 86400.
- The Friday deploy that ate my weekend, and whose fault it actually was (mine)
MOTD (11)
- BBRv3 is in the kernel. CUBIC is still the default. Here is when to care.
- TCP MD5 for BGP authentication is thirty years old. TCP-AO is the replacement. Nobody has migrated.
- cups-browsed held UDP 631 open to the world. Two years after the CVEs, the architecture finally changed.
- DNSSEC validates about 38% of the internet. Here's why the other 62% hasn't moved.
- ML-KEM has been the default SSH key exchange for two years. Enterprise IT just noticed.
- Your curl switched to HTTP/3 without asking. The packet trace shows encrypted UDP.
- Six years of WireGuard in the Linux kernel. I was wrong to be skeptical.
- iptables is a shim now. The shim has edge cases. Rewrite your rules.
- The post-quantum key exchange already happened. You didn't notice. Good.
- Your browser encrypted your DNS queries. It also decided who gets to read them.
- Certificate lifetimes are collapsing to 47 days and your renewal process is a person