$RodHat_

Topic · 17 articles

Tracing & Performance

The dashboard says the box is fine. The box disagrees. Tracing is how you ask the kernel directly instead of trusting whatever an agent decided to sample, and every tool on this page ships with the OS or one package away from it.

Start with strace used properly, as a counter and not a firehose, and the by-hand walk through /proc for a wedged process. Then bpftrace and dtrace for the questions strace can't answer, perf for CPU, and ftrace for when you need the kernel's own call graph. The Tales are what this looks like at 3am.

Start here

Read these first, in this order.

  1. 01

    Tip · 2026-08-13

    strace -c is a profiler. Stop spraying and praying.

    Most people open strace, get buried in output, and close the terminal. Four flags fix that: -c counts, -e filters, -P follows a single path, -T times each call. Here is how strace is actually supposed to work.

  2. 02

    Tip · 2026-08-06

    Read a wedged process with your hands, not a dashboard

    A process is hung and the graphs won't say why. Here's the /proc, ps, strace, and dmesg walk that finds a stuck worker in about ninety seconds, no agent, no vendor, no login.

  3. 03

    Tip · 2026-08-16

    bpftrace gives Linux what FreeBSD had in 2005. The one-liners are worth the wait.

    dtrace showed up on Solaris in 2004, shipped in FreeBSD 7 in 2008, and Linux users spent the next decade pretending strace was sufficient. bpftrace is the real answer, doing dynamic kernel tracing, histograms, stack walks, zero overhead when idle.

  4. 04

    Tip · 2026-07-27

    Get a real disk latency histogram in one line of bpftrace

    iostat gives you an average, and averages hide the tail that's actually hurting you. A four-line bpftrace program prints a log2 histogram of block I/O latency per device, live, on a production box, with no agent and no restart.

  5. 05

    Tip · 2026-07-20

    Find out which syscall is actually eating your latency with one dtrace one-liner

    A real dtrace/bpftrace one-liner for finding the syscall responsible for tail latency, plus why strace -T won't get you there.

  6. 06

    Tip · 2026-08-14

    Your kernel ships a CPU profiler. perf(1) is the key.

    perf stat hands you hardware performance counters (cycles, cache misses, branch mispredictions) in seconds. perf record samples call stacks at full speed. Both are already installed and beat any SaaS APM for understanding CPU-bound problems.

  7. 07

    Tip · 2026-08-22

    ftrace has been on your machine since 2.6.27. Here's how to actually use it.

    ftrace is Linux's built-in kernel function tracer, accessible directly via /sys/kernel/debug/tracing/. No compiler, no LLVM, no kernel headers: write to files, read call graphs. Here's the workflow for function tracing, call-graph timing, IRQ latency, and isolated trace instances.

The syscalls underneath

Version introduced, sourced from the man page. Full cards in the syscall reference, or check your kernel.

InterfaceSinceWhat it does
sched_setaffinity(2)Linux 2.5.8Set the set of CPUs a thread may run on (taskset is the CLI wrapper).
ioprio_set(2)Linux 2.6.13Set a thread's I/O scheduling class and priority (ionice is the CLI wrapper).
futex(2)Linux 2.6.0Fast userspace locking: wait on or wake threads sleeping on a 32-bit word in memory.
rseq(2)Linux 4.18Register a per-thread struct that the kernel keeps updated with the current CPU ID, enabling lock-free per-CPU data updates via restartable critical sections.

More console tips (3)

Rod's tales (3)

MOTD (4)

Other topics