Stop waiting beside tcpdump like it owes you an incident

Intermittent network failures have one dependable property: they stop the moment you attach tcpdump and stare at the terminal.

The application has been resetting connections twice a day for three weeks. You log in, start a capture, and receive six hours of healthy ACKs plus an increasingly personal dislike of Ethernet. You stop the capture to reclaim disk space. Twelve minutes later the failure returns.

Do not run one enormous capture. Run a ring buffer.

Rotate by size before the filesystem files an incident of its own

tcpdump can close a capture file after it reaches a chosen size and continue into the next numbered file:

sudo tcpdump -i eth0 -nn -s 0 \
  -w /var/tmp/app.pcap \
  -C 100 -W 24 \
  'host 192.0.2.44 and tcp port 443'

The important parts:

• -w writes raw packets for later analysis instead of printing summaries.
• -C 100 rotates after roughly 100 million bytes per file.
• -W 24 keeps 24 files and reuses the oldest names.
• -s 0 captures the complete packet instead of an old-fashioned truncated snap length.
• -nn prevents name and service lookups from adding noise and occasionally causing their own traffic.

That gives you a bounded capture set instead of an enthusiastic attempt to fill /var.

The exact file-naming behavior varies slightly among tcpdump versions, so test the command on the host before leaving it unattended. Yes, test the debugging tool. Production has enough improvisational theater already.

Filter before capture, not after disk exhaustion

A broad capture feels safer because you cannot miss traffic you did not filter out. It also records backups, monitoring, SSH sessions, storage replication, and whichever chatty service discovered multicast this week.

Start from the failure’s actual boundary:

'host 192.0.2.44 and tcp port 443'

Or capture traffic between two subnets:

'net 10.24.0.0/16 and net 10.80.0.0/16'

Or exclude the management connection keeping you attached:

'not port 22 and host 192.0.2.44'

Berkeley Packet Filter expressions are applied in the capture path. A good filter reduces disk writes, CPU work, review time, and the chance that your evidence contains credentials or unrelated user traffic.

Do not capture payloads merely because you can. Headers are often sufficient for resets, retransmissions, handshake failures, MTU problems, and timing analysis. If payloads may contain personal data, secrets, or regulated information, involve the people responsible for that data before collecting it.

Time rotation is useful when the incident follows a clock

Many tcpdump builds also support rotating after a time interval with -G:

sudo tcpdump -i eth0 -nn -s 0 \
  -G 300 \
  -w '/var/tmp/app-%Y%m%d-%H%M%S.pcap' \
  'host 192.0.2.44 and tcp port 443'

That starts a new file every five minutes.

Do not assume combining -G, -C, and -W behaves identically across every vendor build. The semantics and filename handling have accumulated historical barnacles. Read the local man page, run a ten-minute test, and inspect the files it actually creates.

Unix documentation is installed on the machine specifically so you do not have to trust a blog written against somebody else’s package version. Including this one.

Preserve the window when the alert fires

A ring buffer is only useful if the evidence survives long enough to inspect.

When monitoring detects the failure, stop or copy the capture set before normal rotation overwrites the relevant files. A basic trigger might do this:

stamp=$(date -u +%Y%m%dT%H%M%SZ)
mkdir -p "/var/tmp/incident-$stamp"
cp -p /var/tmp/app.pcap* "/var/tmp/incident-$stamp/"

Copying live files is imperfect. The currently open capture may still be changing. The cleaner method is to signal the capture process, wait for it to close the file, preserve the set, then restart the capture if needed.

For a service-managed capture, make the stop-and-preserve operation explicit. Do not use pkill tcpdump on a host where somebody else may be collecting different evidence unless you enjoy interdisciplinary incident response.

Read the capture with questions, not vibes

Start with packet counts and conversations:

tshark -r app.pcap -q -z conv,tcp

Look for resets:

tshark -r app.pcap -Y 'tcp.flags.reset == 1'

Look for retransmission analysis flags:

tshark -r app.pcap -Y 'tcp.analysis.retransmission || tcp.analysis.fast_retransmission'

Then align packet timestamps with application logs, load-balancer logs, firewall changes, route transitions, and the actual alert time.

A capture without a timeline becomes a Rorschach test for network engineers. Everyone finds their preferred failure mode.

The boring operational version

For recurring intermittent failures:

1. Define the narrowest useful capture filter.
2. Bound storage with rotation.
3. Run under a service manager with a dedicated directory and permissions.
4. Preserve the ring when monitoring detects the event.
5. Record the trigger timestamp and timezone.
6. Restrict access to the captures.
7. Delete them when the investigation is complete.

The clever part is not the packet decoder. The clever part is arranging to possess the packets from five minutes before anybody knew there was a problem.

Standing beside a live terminal is not observability. It is fishing with a flashlight.