$RodHat_/search
Console Tips

ktrace, kdump and truss: tracing syscalls on FreeBSD without dtrace

Published by

This article may contain Amazon affiliate links. I may earn a commission if you buy through them. Disclosure.

Editorial card: ktrace, kdump and truss: tracing syscalls on FreeBSD without dtrace
Editorial card: RodHat
On this page · 6 sections
  1. truss: the quick one
  2. ktrace: record now, read later
  3. Choosing trace points
  4. Replaying with kdump
  5. Which one when
  6. The cons

Every Linux person who lands on a FreeBSD box types strace and gets “command not found”. Then they assume the platform has no syscall tracer. It has had two for decades, and one of them records to a file so you can read the evidence after the incident instead of during it.

I covered the Linux side in strace without the firehose and the heavy artillery in dtrace syscall latency. This is the middle tier: truss for a quick look, ktrace plus kdump for a recording.

truss: the quick one

truss(1) is the closest thing to strace, and it is fine for a first pass.

truss -f -o /tmp/web.truss -p 4121
truss -f -c ./flaky-script.sh
  • -f follows forks. Forget it and you trace a shell that does nothing but wait.
  • -o writes to a file so the tracee’s own output is not interleaved with yours.
  • -c prints a summary of syscall counts instead of the stream. Run this first. A process that makes 200,000 stat calls to start up is an answer by itself.
  • -s N raises the string length it prints. The default truncates exactly the path or buffer you wanted.

Like strace, truss stops the target at each syscall boundary, so it slows the target down noticeably. Do not leave it attached to a production daemon for a lunch break.

ktrace: record now, read later

ktrace(1) has the kernel append trace records to a file as the process runs. Nothing is decoded on the fly, which is why it is cheaper to leave running, and why you can hand the file to someone else.

ktrace -di -f /tmp/web.ktrace -p 4121
# ... reproduce the problem ...
ktrace -C
  • -p pid attaches to a running process. Without it, ktrace command args starts the command traced.
  • -d also traces future descendants of the process. -i makes children inherit the trace. If the thing you are chasing forks workers, you want both, hence -di.
  • -f names the output. The default is ktrace.out in the current directory, which is a fine way to litter a home directory.
  • -C clears all tracing on the system. -c clears it for the named pid.

Choosing trace points

You select what to record with -t and a string of letters:

ktrace -di -t cni -f /tmp/web.ktrace -p 4121
  • c is syscall entry and return.
  • n is namei, the pathname lookups. This is the one that tells you which file the program tried to open.
  • i is the data buffers moved by read and write.
  • s is signals and w is context switches.
  • u is userland records, which almost nobody uses.

Check ktrace(1) on your release for the exact set. Be stingy with i. Recording the I/O payloads of a busy server fills a disk quickly and makes the file very large, and you probably wanted c and n.

Replaying with kdump

kdump -f /tmp/web.ktrace | less
kdump -E -f /tmp/web.ktrace
kdump -T -f /tmp/web.ktrace

kdump(1) decodes the file. -E prints elapsed timestamps, -R relative ones and -T absolute ones, so a gap between two calls is visible as a number instead of a feeling. Pipe the output through grep -n NAMI to see just the path lookups, and you have the “what files did this actually touch” answer without ever having attached anything interactive.

Because it is a file, the workflow is: record on the sick box, copy the file off, read it somewhere with a decent terminal and your own vi. That matters when the sick box is the one you do not want to keep poking.

Which one when

  • Thirty seconds, no setup: truss -f -c.
  • A flaky thing that happens “sometimes”: ktrace -di, leave it, collect the file after it fails.
  • Latency distributions or system-wide questions: neither. Use dtrace.

For what happens when the tracer is the problem, see the ktrace jail privilege regression write-up. If you want the full FreeBSD toolkit in one place, the FreeBSD and OpenBSD hub has it.

The cons

  • Both tools are per-process. Neither will tell you what the system as a whole is doing.
  • truss slows the target. ktrace is lighter but still not free on a syscall-heavy workload.
  • A ktrace.out file can contain data read and written by the process, including credentials, if you recorded i. Treat the file as sensitive and delete it.
  • -C clears tracing for everything on the machine. If someone else is tracing something, you just ended their session.
  • Output formats differ between releases. Decode a file with the kdump from the same or a newer release than the one that wrote it.

Reach for truss -f -c first, ktrace -di -t cn second, and dtrace when both have told you where to point it.