Your kernel ships a CPU profiler. perf(1) is the key.

Someone is running a SaaS APM in production, paying per host per month, watching pretty charts that say CPU is high. The charts do not tell them which function is high, because that costs more. Somewhere on that same host, in /usr/bin/, perf is sitting unused.

This is the situation.

perf is the Linux kernel’s performance counter interface dressed up as a user-space tool. It talks to the Performance Monitoring Unit (PMU) built into your CPU — real hardware counters, not samples from a userland timer. The same hardware Intel and AMD use for their own profiling tools. It ships in linux-tools-$(uname -r) on Debian-based distros, perf on Arch. On FreeBSD, dtrace is the equivalent. On Linux, it’s one package away.

perf stat: hardware counters in 10 seconds

Run a command, get counters:

perf stat ./myapp

Output:

 Performance counter stats for './myapp':

          8,423.21 msec task-clock                #    0.998 CPUs utilized
               142      context-switches          #   16.860 /sec
                 3      cpu-migrations            #    0.356 /sec
             1,204      page-faults               #  142.935 /sec
    28,941,432,105      cycles                    #    3.437 GHz
    18,653,211,887      instructions              #    0.64  insn per cycle
     3,204,112,002      branches                  #  380.412 M/sec
       342,091,445      branch-misses             #   10.68% of all branches

insn per cycle (IPC) is the efficiency number. Modern CPUs can theoretically execute 4+ instructions per cycle with superscalar execution. 0.64 IPC means the CPU is spending most of its time waiting — waiting for memory, waiting for branch resolution, pipeline stalled.

branch-misses at 10.68% is high. Branch mispredictions cause the CPU to flush its pipeline and restart. Above 5% starts hurting on tight loops; 10%+ is a performance problem that has a source-level fix.

Attach to a running process and sample for 10 seconds:

perf stat -p $pid sleep 10

Specific events if you care about cache behavior:

perf stat -e cycles,instructions,cache-misses,cache-references,branch-misses ./myapp

cache-misses / cache-references gives you the L1 cache miss rate. High miss rate means memory-bound, not CPU-bound. Different problem, different fix. perf stat tells you which category you’re in before you spend three hours reading source code.

perf record + perf report: where the CPU time goes

perf stat tells you that you’re burning cycles. perf record tells you where:

perf record ./myapp
perf report

perf record samples the instruction pointer at 1000 Hz by default and writes to perf.data. perf report opens an ncurses browser of your call tree sorted by % CPU time.

For a running process:

perf record -p $pid sleep 30
perf report

Higher sampling frequency for finer resolution:

perf record -F 9999 ./myapp

More frequency means more overhead, but still nothing close to strace’s ptrace penalty. perf samples asynchronously via NMI interrupts; the process keeps running at near-full speed.

--stdio if you want stdout over the TUI:

perf report --stdio | head -60

Call graphs: who called the hot function

By default perf report shows a flat profile — which function is hot. Useful. Not complete. The full picture requires call graphs: who is calling the hot function?

Three options, in order of ease:

# Frame pointers. Fast, minimal runtime overhead.
# Requires binaries compiled with -fno-omit-frame-pointer.
# Most distro packages strip frame pointers. This is a known pain.
perf record --call-graph fp ./myapp

# DWARF unwinding. Works without frame pointers if debug symbols exist.
# High per-sample overhead — drop frequency to compensate.
perf record -F 99 --call-graph dwarf ./myapp

# Intel Last Branch Record (LBR). Hardware-assisted, near-zero overhead.
# Limited stack depth (32 levels on most CPUs). Intel only, not AMD.
perf record --call-graph lbr ./myapp

The frame pointer problem is real and tedious. glibc, libstdc++, and most production-compiled applications omit frame pointers for the performance they trade. Your options: rebuild with -fno-omit-frame-pointer and accept the minor regression, use DWARF mode with debug packages, or use LBR on Intel hardware. On containers, the practical move is to run perf record --call-graph dwarf at low frequency against the host PID — find it in /proc rather than from inside the container.

perf annotate: instruction-level

After perf report, press a on any symbol to get annotated source — each line decorated with the percentage of samples that landed there. No source? Assembly. Either way, you see exactly which instruction is hot.

From the command line:

perf annotate --symbol=my_hot_function --stdio

When perf report says a function consumes 40% of CPU time and you can’t see why from the source, annotate shows you whether it’s a tight inner loop, a mispredicted branch, or a cache line boundary crossing. At that point you’re rewriting the loop, adjusting alignment, or adding a prefetch hint — and you know it from evidence.

perf top: live view, no perf.data

sudo perf top

Real-time profile of the entire system. Every process, every function, sorted by current hot samples. When top(1) shows CPU at 100% and you don’t know which process or function, perf top narrows the field faster than anything else. No file written, no postprocessing step.

For one process:

perf top -p $pid

The paranoid wall

Most Linux systems have /proc/sys/kernel/perf_event_paranoid set to 2 or higher:

cat /proc/sys/kernel/perf_event_paranoid

Value	What you can do
2	Sample your own processes only. No kernel symbols.
1	Kernel symbols visible without root.
0	Raw CPU performance data for all processes without root.
-1	No restrictions.

You can profile your own processes at paranoid=2. For kernel symbols or cross-process profiling, either run as root, set paranoid=1 temporarily, or use CAP_PERFMON — the dedicated capability added in kernel 5.8 that replaced the blunter CAP_SYS_ADMIN for this purpose.

In containers: most runtimes drop CAP_PERFMON by default. Either run perf on the host against the process’s host PID, or add --cap-add CAP_PERFMON to your container launch. nsenter lets you enter the container’s namespaces from the host while keeping the host’s capabilities — you can run the host’s perf against a container PID without the container having anything installed.

When perf is the wrong answer

perf tracks CPU time. It does not help you when:

• The process is blocking — sleeping in a read(), epoll_wait(), or a futex. perf shows almost nothing because the CPU is idle. Use strace -T to find which syscall it’s blocking in, then bpftrace for low-overhead production confirmation.
• Syscall frequency is the problem — perf can count syscall events, but strace -c gives you the table faster with less ceremony.
• You need heap allocation patterns — Valgrind/Massif or perf mem for that territory.

If perf stat shows high IPC and low cache miss rate and the process is still slow, the answer might be lock contention. perf lock analyzes lock acquisition patterns. It’s fiddly to set up — requires CONFIG_LOCKDEP in the kernel — but it’s the right tool when perf report shows significant time in futex-related frames.

The distinction: CPU-bound → perf. I/O-bound or lock-bound → other tools. perf stat IPC + cache miss rate tells you which side you’re on in under 30 seconds.

---

The hardware counter model — what the PMU can actually measure, how sampling interacts with pipeline stalls, why IPC below 1.0 means what it means — is covered properly in the architecture chapters of Operating Systems: Three Easy Pieces. The section on memory hierarchy explains why cache-miss rate matters more than raw clock speed for most modern workloads. It’s the framework that makes the perf numbers readable rather than just decorative.