ftrace has been on your machine since 2.6.27. Here's how to actually use it.
Published by RodHat
Every time someone reaches for bpftrace to answer “what is the kernel doing right now,” I think about the year 2008, when Linux 2.6.27 shipped with a built-in function tracer that requires zero external dependencies. You write to files. You read from files. The tool is already there.
bpftrace is excellent. I use it. But it requires a working LLVM, kernel headers, and a BPF JIT-capable kernel. On the machine that’s on fire, at least one of those is broken. ftrace works on everything: containers, embedded systems, ancient kernels, VMs where you forgot to install the header package. If it’s Linux 2.6.27 or later, it has ftrace.
Getting there
debugfs is usually mounted at boot. Check first:
mount | grep debugfs
# debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)If it’s not there:
mount -t debugfs none /sys/kernel/debugThe tracing interface lives at /sys/kernel/debug/tracing/. The files that matter:
available_tracers — tracers compiled into this kernel
current_tracer — active tracer; write to switch
tracing_on — 1 runs, 0 pauses (ring buffer keeps allocating, stops filling)
trace — snapshot of the ring buffer
trace_pipe — streaming read; blocks like tail -f, consumes as it reads
set_ftrace_filter — which functions to trace (glob patterns, one per line)
set_ftrace_notrace — exclusions (same syntax)
available_filter_functions — every traceable kernel symbol
set_ftrace_pid — restrict to a specific PID
trace_marker — write a string here from userspace; it appears in the trace inline
max_graph_depth — limit call graph depth for function_graphThe function tracer
The simplest tracer. Every kernel function call, timestamped, with the caller:
# 1. Set the tracer
echo function > /sys/kernel/debug/tracing/current_tracer
# 2. Filter — without this, the buffer fills in milliseconds on a busy system
echo 'tcp_*' > /sys/kernel/debug/tracing/set_ftrace_filter
# 3. Enable
echo 1 > /sys/kernel/debug/tracing/tracing_on
# 4. Do the thing you're investigating
# (run a curl, send data, trigger the codepath)
# 5. Stop
echo 0 > /sys/kernel/debug/tracing/tracing_on
# 6. Read
cat /sys/kernel/debug/tracing/trace | head -60Output:
# tracer: function
#
# _-----=> irqs-off
# / _----=> need-resched
# | / _---=> hardirq/softirq
# || / _--=> preempt-depth
# ||| /
# TASK-PID CPU# |||| TIMESTAMP FUNCTION
curl-8924 [002] .... 1042.349123: tcp_sendmsg <-sock_sendmsg
curl-8924 [002] .... 1042.349130: tcp_send_mss <-tcp_sendmsg
curl-8924 [002] .... 1042.349145: tcp_current_mss <-tcp_send_mssTASK-PID is the calling process. TIMESTAMP is seconds from boot. function <-caller is the call site. The four-character field between CPU# and TIMESTAMP is the tracing flags: d means preempt-depth > 0, h means you’re in a hardware interrupt handler, s is softirq. .... means none of the above.
function_graph: call trees with timing
The function tracer gives you a flat stream. function_graph gives you the call tree with entry and exit timestamps — so you get actual durations per function and can see which branch in a call chain is slow:
echo function_graph > /sys/kernel/debug/tracing/current_tracer
# Root of the graph — trace this function and everything it calls
echo 'tcp_sendmsg' > /sys/kernel/debug/tracing/set_graph_function
# Limit depth or you're reading until next Tuesday
echo 4 > /sys/kernel/debug/tracing/max_graph_depth
echo 1 > /sys/kernel/debug/tracing/tracing_on
# trigger the codepath
echo 0 > /sys/kernel/debug/tracing/tracing_on
cat /sys/kernel/debug/tracing/traceOutput:
# tracer: function_graph
#
# CPU DURATION FUNCTION CALLS
# | | | | | | |
1) | tcp_sendmsg() {
1) 0.420 us | lock_sock_nested();
1) | tcp_sendmsg_locked() {
1) 0.183 us | tcp_rate_check_app_limited();
1) | sk_stream_alloc_skb() {
1) 0.091 us | kmem_cache_alloc_node();
1) 0.512 us | }
1) 0.874 us | tcp_push();
1) + 2.456 us | }
1) + 3.312 us | }The markers: + means the call took more than 10µs, ! is more than 100µs, # is more than 1ms. You find the slow branch by scanning for the markers, not by reading every line. That’s the point.
Filtering with set_ftrace_filter
Glob patterns. One pattern per line. > to replace, >> to append:
# Trace only tcp_ functions
echo 'tcp_*' > /sys/kernel/debug/tracing/set_ftrace_filter
# Add ext4_ to the existing filter
echo 'ext4_*' >> /sys/kernel/debug/tracing/set_ftrace_filter
# See what's currently set
cat /sys/kernel/debug/tracing/set_ftrace_filter
# Clear the filter entirely (traces everything — be careful on a loaded box)
echo > /sys/kernel/debug/tracing/set_ftrace_filterIf a function doesn’t appear in the trace after filtering to it by name, check available_filter_functions:
grep '^ext4_write_begin$' /sys/kernel/debug/tracing/available_filter_functionsSome functions can’t be traced — they’re inlined into their callers, or marked notrace in the kernel source (usually because tracing them would cause infinite recursion in the tracing infrastructure itself). If it’s not in available_filter_functions, you can’t filter to it.
You can also filter to a specific process:
echo $$ > /sys/kernel/debug/tracing/set_ftrace_pid # current shell
echo 0 > /sys/kernel/debug/tracing/set_ftrace_pid # clear — trace all pids againirqsoff: finding latency spikes
This is the tracer most people never touch and should. irqsoff records the worst-case period with interrupts disabled during the capture window and gives you the full call graph at that moment:
echo irqsoff > /sys/kernel/debug/tracing/current_tracer
echo 1 > /sys/kernel/debug/tracing/tracing_on
# Run your latency-sensitive workload
# The tracer continuously updates with the worst seen; it doesn't stop on first hit
echo 0 > /sys/kernel/debug/tracing/tracing_on
cat /sys/kernel/debug/tracing/traceOutput starts with the verdict:
# irqsoff latency trace v1.1.5 on 6.8.0-45-generic
# --------------------------------------------------------------------
# latency: 142 us, #4/4, CPU#1 | (M:preempt VP:0, KP:0, SP:0 HP:0 #P:4)
# -----------------
# | task: swapper/1-0 (uid:0 nice:0 policy:0 rt_prio:0)
# -----------------
# => started at: native_queued_spin_lock_slowpath
# => ended at: native_queued_spin_lock_slowpathThen the full call trace at the worst-case point. 142µs is your worst interrupt latency for whatever happened during the capture. preemptoff does the same for preemption disabled. preemptirqsoff catches the worst of either.
This is how you diagnose real-time latency issues without a specialized kernel build. The tracer is already there.
Trace instances: isolated sessions
If you’re running multiple trace sessions — or if a tool like perf ftrace is already using the global context — instances give you completely independent control files and ring buffers:
mkdir /sys/kernel/debug/tracing/instances/tcp-debug
echo function > /sys/kernel/debug/tracing/instances/tcp-debug/current_tracer
echo 'tcp_*' > /sys/kernel/debug/tracing/instances/tcp-debug/set_ftrace_filter
echo 1 > /sys/kernel/debug/tracing/instances/tcp-debug/tracing_on
# ... run the workload ...
cat /sys/kernel/debug/tracing/instances/tcp-debug/trace
echo 0 > /sys/kernel/debug/tracing/instances/tcp-debug/tracing_on
rmdir /sys/kernel/debug/tracing/instances/tcp-debugThe global trace is untouched. Creating an instance creates the full directory structure under it — same files, same semantics, independent state. rmdir cleans it up (the kernel removes the contents when the directory is empty).
Marking userspace events inline
trace_marker lets you write arbitrary strings from userspace that appear as events in the trace timeline. Useful for correlating application behavior with kernel activity:
echo 1 > /sys/kernel/debug/tracing/tracing_on
echo "before: connection attempt" > /sys/kernel/debug/tracing/trace_marker
curl -s https://example.com -o /dev/null
echo "after: connection complete" > /sys/kernel/debug/tracing/trace_marker
echo 0 > /sys/kernel/debug/tracing/tracing_on
cat /sys/kernel/debug/tracing/trace | grep -E 'trace_marker|tcp_connect'The markers appear with the same timestamp format as kernel events, so you can see exactly where in the kernel trace your userspace event landed.
Cleanup
ftrace state persists until you reset it. The next person who touches the system — possibly you, three hours later — inherits your tracer settings and wonders why performance is slightly wrong.
Always clean up when done:
echo nop > /sys/kernel/debug/tracing/current_tracer
echo > /sys/kernel/debug/tracing/set_ftrace_filter
echo > /sys/kernel/debug/tracing/set_ftrace_notrace
echo > /sys/kernel/debug/tracing/set_graph_function
echo > /sys/kernel/debug/tracing/trace # clear ring buffer
echo 1 > /sys/kernel/debug/tracing/tracing_on # leave ready but tracing nopcurrent_tracer: nop is the null tracer — it’s enabled but does nothing. The ring buffer overhead when running nop is negligible. The overhead of forgetting to reset function with no filter on a busy system is not.
ftrace vs bpftrace: when to use which
bpftrace wins for: per-event aggregation, histograms, maps, userspace probes, any analysis that requires computation rather than a raw trace dump.
ftrace wins for: call-graph timing with function_graph (cleaner output than bpftrace for this), IRQ and preemption latency tracing (those tracers simply don’t exist in bpftrace), environments where BPF JIT is disabled or restricted, kernels too old for bpftrace, and cases where you want to leave a trace running for hours and read it once — the ring buffer approach is simpler than a long-running bpftrace script.
And: ftrace has been here since 2008. There is no Linux machine you will ever touch that doesn’t have it. That is the argument.
For the kernel internals behind the trace output — how the ring buffer works, what the flag characters mean, how irqsoff computes its latency measurement — Documentation/trace/ftrace.rst in the kernel source tree is authoritative and actually readable. For the full Linux tracing ecosystem (kprobes, uprobes, tracepoints, perf events, eBPF, ftrace) and how they fit together, Brendan Gregg’s BPF Performance Tools covers it from end to end. For the kernel subsystems you’ll be tracing into — scheduler, memory allocator, VFS, TCP stack — Kerrisk’s The Linux Programming Interface provides the context that makes the call graphs legible. Internal to this site: the bpftrace one-liners tip covers the BPF side of the same toolbox, and the perf stat and record tip covers the hardware counter and sampling side.