bpftrace gives Linux what FreeBSD had in 2005. The one-liners are worth the wait.

I’ve been running dtrace on FreeBSD since before most people reading this knew what a probe was. It was — and still is — one of the most important observability tools ever written. Solaris shipped it in 2004. FreeBSD had it by 2008. Linux users spent the better part of a decade with strace duct-taped to a production box, wondering why there was blood on the floor.

bpftrace is not dtrace. The data model is different, the language is different, the probe namespace is different. But it does what dtrace does: it lets you instrument the running kernel and userspace dynamically, with no kernel recompile, no reboot, and no overhead when your probes aren’t firing. The Linux kernel grew eBPF into something powerful enough to support real tracing, and bpftrace builds the D-language-esque surface on top of it. Grudgingly: it’s good. Ungrudgingly: some of the one-liners are cleaner than equivalent dtrace.

Install it — on most distributions the package is just bpftrace. You need kernel 4.9+ for basic kprobes; 5.8+ for everything worth doing. Check what you have with bpftrace --info.

Probe types: the vocabulary before the one-liners

bpftrace has four probe types you’ll use constantly and two you’ll use rarely:

• kprobe:function_name — fires at the entry of a kernel function. kretprobe:function_name fires at return. These are dynamic — any exported kernel symbol.
• tracepoint:subsystem:event — fires at a kernel static tracepoint. More stable ABI than kprobes across kernel versions. tracepoint:syscalls:sys_enter_openat is an example.
• uprobe:/path/to/binary:function — fires at the entry of a userspace function. uretprobe for the return. Works on any ELF binary with symbols.
• software:event:count and hardware:event:count — PMU-backed events. software:faults:1 fires on every page fault. hardware:cache-misses:100 fires every 100 cache misses.

Built-in variables you’ll use in every script: pid, tid, comm (process name, 16 chars), args (the probe’s argument struct), retval (kretprobe/uretprobe only), nsecs (wall clock in nanoseconds), kstack and ustack (kernel/user stack traces), curtask (the task_struct pointer, for when you need to dig into kernel internals).

Maps: @name[key] = value stores aggregated data. @name = hist(expr) builds a log2 histogram. @name = count() counts. They flush to stdout at program exit unless you print() them explicitly.

Which files is this process opening?

bpftrace -e 'tracepoint:syscalls:sys_enter_openat /comm == "nginx"/ {
  printf("%s\n", str(args->filename));
}'

The /comm == "nginx"/ is a filter — it fires only for processes named nginx. Remove it to see everything on the box, which will scroll faster than you can read in any interesting environment.

For the filename, args->filename is a userspace pointer; str() copies it into the BPF stack. You need str() — raw pointers from userspace aren’t readable directly in BPF context.

Want the full path with process and pid:

bpftrace -e 'tracepoint:syscalls:sys_enter_openat {
  printf("%d %s %s\n", pid, comm, str(args->filename));
}'

This replaces strace -e openat -p $pid for most “what is this thing reading” questions, and unlike strace it attaches to all processes simultaneously with negligible overhead. strace is still useful when you need the full argument dump and return value correlation — but for file tracing at scale, bpftrace wins.

Disk I/O latency histogram

The question you actually want answered is not “is there disk I/O” — iostat tells you that. The question is “what is the distribution of I/O latency, and are there outliers killing tail latency.”

bpftrace -e '
tracepoint:block:block_rq_issue {
  @start[args->dev, args->sector] = nsecs;
}
tracepoint:block:block_rq_complete {
  $s = @start[args->dev, args->sector];
  if ($s > 0) {
    @latency_us = hist((nsecs - $s) / 1000);
    delete(@start[args->dev, args->sector]);
  }
}'

Let it run for 10–30 seconds under load, then Ctrl-C. Output:

@latency_us:
[0]                    4 |                                                    |
[1]                  312 |@@@@@                                               |
[2, 4)              1841 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                      |
[4, 8)              2109 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@                |
[8, 16)              981 |@@@@@@@@@@@@@@@@                                    |
[16, 32)             412 |@@@@@@@                                             |
[32, 64)             199 |@@@                                                 |
[64, 128)             44 |                                                    |
[128, 256)            18 |                                                    |
[256, 512)             3 |                                                    |
[512, 1K)              1 |                                                    |

The log2 histogram shows most I/Os completing in 4–16µs (NVMe local) with a long tail out to 512µs. If you see a bimodal distribution — cluster at 4µs and another cluster at 2ms — that’s a hardware or queue depth issue. The histogram shows it; iostat’s average hides it.

Top syscalls by process

Who is hammering read(2) right now:

bpftrace -e 'tracepoint:syscalls:sys_enter_read { @[comm, pid] = count(); }
  interval:s:5 { print(@); clear(@); }'

interval:s:5 fires every 5 seconds regardless of syscall activity. print(@) dumps the current aggregation; clear(@) resets it. Hit Ctrl-C to stop. You get a rolling top-read-callers view without any per-call output overhead.

Same pattern works for write, sendmsg, recvmsg, stat — substitute the tracepoint name. bpftrace -l 'tracepoint:syscalls:*' lists all available syscall tracepoints on your kernel.

malloc call-size distribution in a running process

This is where uprobe earns its keep. You don’t recompile, you don’t restart, you don’t need debug symbols for libc:

bpftrace -e 'uprobe:/lib/x86_64-linux-gnu/libc.so.6:malloc /pid == 4821/ {
  @alloc_bytes = hist(arg0);
}'

arg0 through arg9 are the positional arguments to the probed function. malloc(size_t size) — arg0 is size. The histogram shows the distribution of allocation sizes in the live process.

If you’re chasing a leak and want to correlate allocation size with call site:

bpftrace -e 'uprobe:/lib/x86_64-linux-gnu/libc.so.6:malloc /pid == 4821/ {
  @[ustack] = hist(arg0);
}'

That maps each unique user-space call stack to a histogram of allocation sizes. It will slow the target process down meaningfully — uprobe with stack walks is not zero-cost. Use on a dev box or under controlled conditions.

The libc path varies by distribution and architecture. On musl systems (Alpine) it’s /lib/ld-musl-x86_64.so.1 or similar. ldd /path/to/binary shows what libc your target links against.

Page fault sources with kernel stacks

bpftrace -e 'software:faults:1 /pid == 4821/ { @[kstack] = count(); }'

Runs for as long as you let it, accumulates kernel stack traces at every page fault in the target process. At exit it prints stacks ranked by frequency. Useful for distinguishing anonymous mmap faults from file-backed faults, and for confirming whether demand paging is happening where you think it is.

For user stacks instead:

bpftrace -e 'software:faults:1 /pid == 4821/ { @[ustack] = count(); }'

software:faults:1 means “fire on every page fault” — the :1 is the sample period. software:faults:100 would sample 1-in-100, reducing overhead at the cost of less accurate counts.

Kernel function latency (kprobes)

Any exported kernel function. Say you want to know how long vfs_read is taking:

bpftrace -e '
kprobe:vfs_read { @start[tid] = nsecs; }
kretprobe:vfs_read /@start[tid]/ {
  @latency_us = hist((nsecs - @start[tid]) / 1000);
  delete(@start[tid]);
}'

@start[tid] uses the thread ID as key so concurrent threads don’t stomp each other. The filter /@start[tid]/ on the return probe skips threads that entered before the probe attached.

kprobe events use arg0–arg9 for arguments (in the order they appear in the function signature). kretprobe events expose retval. Finding the argument order: bpftrace -lv 'kprobe:vfs_read' shows the argument names on kernels compiled with BTF (5.8+ with CONFIG_DEBUG_INFO_BTF=y, which includes most modern distributions). Without BTF you’re reading the kernel source.

The thing strace can’t do at this scale

strace uses ptrace(2) — it stops the process at every syscall boundary. For a single-process debug session that’s fine. For profiling under production load, the overhead is prohibitive and attaching to multiple processes simultaneously is awkward.

bpftrace’s overhead is bounded by the BPF verifier and map operations, not by process-stop/resume cycles. A kprobe with a count aggregation costs on the order of 100–300 nanoseconds per hit. A kprobe with a hist() and a stack walk is closer to 1–5 microseconds. For syscall-heavy processes this adds up, but it’s a fraction of the strace tax, and it applies globally across all processes at once.

The practical limit: uprobe on a hot path (tight malloc loop, high-frequency timer) will cost you. Everything else, you can leave running for minutes without noticing on [perf stat](/tips/2026-08-14-perf-stat-record/).

Running without root

bpftrace requires CAP_BPF and CAP_PERFMON (Linux 5.8+). On older kernels it required full root. You can grant these capabilities to a specific binary:

setcap cap_bpf,cap_perfmon+eip /usr/bin/bpftrace

Or add your user to a group that has these capabilities via udev rules. On production servers I prefer a wrapper that grants the minimum capabilities to the bpftrace binary for a specific invocation rather than setcap on the binary permanently — but that’s a policy call for your environment.

---

The dtrace documentation from the Solaris days is still worth reading for the mental model — the probe model, aggregations, and speculation semantics are identical in concept even where the syntax differs. On Linux, the authoritative reference for eBPF internals (which bpftrace compiles down to) is The Linux Programming Interface — Kerrisk’s treatment of the BPF syscall and verifier is the clearest published explanation of why the BPF sandbox works and where its limits are. For understanding what you’re actually tracing at the kernel level, you need Advanced Programming in the UNIX Environment — Stevens on file descriptors, VFS, and the process model is the foundation that makes the tracepoint names make sense.