Every Linux person who lands on a FreeBSD box types strace and gets “command not found”. Then they assume the platform has no syscall tracer. It has had two for decades, and one of them records to a file so you can read the evidence after the incident instead of during it.
I covered the Linux side in strace without the firehose and the heavy artillery in dtrace syscall latency. This is the middle tier: truss for a quick look, ktrace plus kdump for a recording.
truss: the quick one
truss(1) is the closest thing to strace, and it is fine for a first pass.
truss -f -o /tmp/web.truss -p 4121
truss -f -c ./flaky-script.sh
-ffollows forks. Forget it and you trace a shell that does nothing but wait.-owrites to a file so the tracee’s own output is not interleaved with yours.-cprints a summary of syscall counts instead of the stream. Run this first. A process that makes 200,000statcalls to start up is an answer by itself.-s Nraises the string length it prints. The default truncates exactly the path or buffer you wanted.
Like strace, truss stops the target at each syscall boundary, so it slows the target down noticeably. Do not leave it attached to a production daemon for a lunch break.
ktrace: record now, read later
ktrace(1) has the kernel append trace records to a file as the process runs. Nothing is decoded on the fly, which is why it is cheaper to leave running, and why you can hand the file to someone else.
ktrace -di -f /tmp/web.ktrace -p 4121
# ... reproduce the problem ...
ktrace -C
-p pidattaches to a running process. Without it,ktrace command argsstarts the command traced.-dalso traces future descendants of the process.-imakes children inherit the trace. If the thing you are chasing forks workers, you want both, hence-di.-fnames the output. The default isktrace.outin the current directory, which is a fine way to litter a home directory.-Cclears all tracing on the system.-cclears it for the named pid.
Choosing trace points
You select what to record with -t and a string of letters:
ktrace -di -t cni -f /tmp/web.ktrace -p 4121
cis syscall entry and return.nis namei, the pathname lookups. This is the one that tells you which file the program tried to open.iis the data buffers moved byreadandwrite.sis signals andwis context switches.uis userland records, which almost nobody uses.
Check ktrace(1) on your release for the exact set. Be stingy with i. Recording the I/O payloads of a busy server fills a disk quickly and makes the file very large, and you probably wanted c and n.
Replaying with kdump
kdump -f /tmp/web.ktrace | less
kdump -E -f /tmp/web.ktrace
kdump -T -f /tmp/web.ktrace
kdump(1) decodes the file. -E prints elapsed timestamps, -R relative ones and -T absolute ones, so a gap between two calls is visible as a number instead of a feeling. Pipe the output through grep -n NAMI to see just the path lookups, and you have the “what files did this actually touch” answer without ever having attached anything interactive.
Because it is a file, the workflow is: record on the sick box, copy the file off, read it somewhere with a decent terminal and your own vi. That matters when the sick box is the one you do not want to keep poking.
Which one when
- Thirty seconds, no setup:
truss -f -c. - A flaky thing that happens “sometimes”:
ktrace -di, leave it, collect the file after it fails. - Latency distributions or system-wide questions: neither. Use dtrace.
For what happens when the tracer is the problem, see the ktrace jail privilege regression write-up. If you want the full FreeBSD toolkit in one place, the FreeBSD and OpenBSD hub has it.
The cons
- Both tools are per-process. Neither will tell you what the system as a whole is doing.
trussslows the target.ktraceis lighter but still not free on a syscall-heavy workload.- A
ktrace.outfile can contain data read and written by the process, including credentials, if you recordedi. Treat the file as sensitive and delete it. -Cclears tracing for everything on the machine. If someone else is tracing something, you just ended their session.- Output formats differ between releases. Decode a file with the
kdumpfrom the same or a newer release than the one that wrote it.
Reach for truss -f -c first, ktrace -di -t cn second, and dtrace when both have told you where to point it.
