Stop guessing what's listening on that port

Somebody says “port 8080 is in use.” Half the industry’s response is netstat -an | grep 8080, which confirms the thing you already knew and tells you nothing you needed. A socket is bound. Congratulations. By what?

The one flag combination worth memorizing

Linux:

ss -tlnp

-t TCP, -l listening only, -n numeric (don’t make me wait on reverse DNS for 40 sockets), -p show the process. That last one needs root or you get a polite blank column.

FreeBSD:

sockstat -4 -6 -l

Same idea, better default output — user, command, PID, and the local address in one table without asking.

Learn one of these properly and you never type netstat again. netstat on Linux has been reading /proc/net/tcp — a text file it re-parses line by line — since the Clinton administration. ss pulls the same data out of the kernel over netlink, which is why it doesn’t visibly stall on a box with 200,000 sockets. On a busy box that difference is minutes.

The column everyone ignores

Run ss -tln and look at what comes back:

State   Recv-Q  Send-Q  Local Address:Port  Peer Address:Port
LISTEN  0       511     0.0.0.0:80          0.0.0.0:*
LISTEN  129     128     0.0.0.0:8080        0.0.0.0:*

On a listening socket, those two queue columns do not mean what they mean on an established one. Send-Q is the accept-queue limit — the backlog your application passed to listen(2), capped by net.core.somaxconn. Recv-Q is how many completed connections are sitting in that queue right now, waiting for the application to call accept().

Which means the second line up there is a running incident. That process asked for a backlog of 128 and has 129 connections stacked up waiting to be accepted. It is not accepting fast enough. Every connection arriving now is getting dropped or, worse, silently retried by the client — and your application log says nothing at all, because from the app’s point of view nothing happened. It never saw those connections.

Confirm it:

nstat -az TcpExtListenOverflows TcpExtListenDrops

If ListenOverflows is climbing, you have found your “random timeouts” bug and it was never the network. It’s a thread pool that’s too small, or a worker blocked on something slow, or a backlog value someone copy-pasted out of a 2009 tutorial.

Who is on the other end

Listening sockets are half the question. When something is hammering you:

ss -tn state established '( dport = :443 or sport = :443 )'

ss has a real filter language and almost nobody uses it. You can ask for exactly the connections you care about instead of grepping a wall of text:

ss -tn state time-wait | wc -l          # drowning in TIME_WAIT?
ss -tn state syn-recv                    # half-open — someone's SYN flooding you
ss -tni dst 10.2.0.0/16                  # everything to one subnet, with TCP internals

That -i is the good one. It dumps the kernel’s per-connection TCP state — congestion window, RTT, retransmits:

cubic wscale:7,7 rto:312 rtt:108.5/12.25 cwnd:10 bytes_retrans:284120 retrans:0/312

An rtt of 108ms to a host in the same rack is a physical problem. A retrans count that grows while you watch is a physical problem. This is the data that ends the twenty-minute argument about whether “the network is fine” — and you got it without opening a single dashboard.

The deleted-socket trap

One more. You kill the process, the port stays bound, and you start suspecting ghosts. Two real causes, and they are not the same:

TIME_WAIT on a socket the server closed first — the kernel holds the tuple for 2×MSL so a stray packet from the old connection can’t land in a new one. That’s correct behavior. SO_REUSEADDR in the application is the fix; disabling tcp_tw_recycle folklore from a 2013 blog post is not (it was removed from Linux in 4.12 because it broke NAT).

A forked child still holding the listening fd. The parent is gone, the child inherited the socket, and the child is not what you grepped for. ss -tlnp shows you the PID that actually holds it. So does:

fuser -n tcp 8080
lsof -i :8080

Kill that one. The port has never once been haunted.

The point

netstat -an | grep answers a question you didn’t ask. Thirty seconds with ss -tlnp tells you the process, the user, the backlog, and whether the accept queue is already on fire — and that last one is a class of outage that never shows up in application logs at all. It shows up as customers saying the site is slow while every graph you own is green.