strace -c is a profiler. Stop spraying and praying.

The typical strace session: attach to a process, get 40,000 lines of output, ^C, grep for something, find nothing, give up and add more logging. This is the wrong mental model. strace has a counting mode. It has syscall group filters. It has path-specific tracing. Most people have never used any of them.

-c: the counting mode nobody uses

Run the process to completion and print a summary table — which syscalls ran, how many times, total time, time per call, and error count:

strace -c ls /usr/local/

Output looks like:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 61.23    0.000412          51         8           getdents64
 18.44    0.000124           7        16           stat
  9.71    0.000065           4        16           close
  ...

getdents64 dominates because you asked it to list a directory. That’s expected. If you ran this against an application and read was consuming 60% of the time, that’s a lead. If stat is running 10,000 times on startup, that’s pathological and you didn’t know until now.

For a running process:

strace -cp $pid

Attach, let it run for 30 seconds under load, ^C, read the table. You will know which syscalls are burning time. This costs less setup time than spinning up a profiler and gives you signal faster. It’s not a replacement for perf or dtrace — it has ptrace(2) overhead, which we’ll get to — but it’s the fastest way to narrow the problem space.

-e: surgical filters instead of fire hose

-e trace= takes individual syscall names or group aliases. The group aliases are the part worth memorizing:

Group	What it covers
%file	Anything that takes a filesystem path: open, openat, stat, lstat, unlink, rename, chmod, access…
%network	Socket operations: socket, connect, bind, listen, sendto, recvfrom, getsockopt…
%process	fork, clone, execve, wait4, exit_group, getpid…
%signal	sigaction, kill, rt_sigreturn, pause…
%ipc	Shared memory, semaphores, message queues
%desc	fd operations: read, write, close, dup, select, poll, epoll_wait, fcntl…
%memory	mmap, mprotect, munmap, brk, madvise…

Making unexpected network connections? Find them:

strace -e trace=%network -p $pid

Something hammering the filesystem at startup?

strace -e trace=%file ./myapp 2>&1 | head -100

Want both network and specific file ops:

strace -e trace=%network,openat,stat ./myapp

-c and -e compose cleanly. Count only the network syscalls:

strace -c -e trace=%network ./myapp

-P: trace one path only

You know the file. You don’t know who’s touching it or when.

strace -P /etc/hosts ./myapp

Every syscall that names /etc/hosts as an argument gets printed. Everything else is silent. If something is opening /etc/shadow when it shouldn’t be, this makes it obvious. If your application is stat-ing a config file on every request instead of caching it at startup, you’ll see a thousand stat("/etc/myapp.conf"...) calls where you expected one.

You can stack multiple -P arguments:

strace -P /etc/passwd -P /etc/shadow ./myapp

-T: time each call

Add wall-clock time to every syscall line:

strace -T -e trace=%network,read -p $pid

Each line ends with <0.003421> — the time in seconds that syscall took. When you see:

read(9, "", 4096)                       = 0 <30.002114>

That’s a read() that blocked for 30 seconds. You’ve found your hang. The fd number and return value tell you what it was reading from. /proc/$pid/fd/9 tells you what fd 9 is attached to right now, or was at process start:

ls -la /proc/$pid/fd/9

-T is the fastest way to correlate “process is slow” with “which syscall it’s blocked in” without attaching a profiler or reading source code.

Following forks

By default strace drops child processes when the parent forks. -f follows them:

strace -f ./myapp

The output is interleaved and annotated with [pid XXXXX] prefixes. Fine for quick checks. For anything with more than a couple workers it becomes unreadable.

-ff -o /tmp/trace writes one file per process:

strace -ff -o /tmp/trace ./myapp

You get /tmp/trace.12345, /tmp/trace.12346, etc. Each worker’s syscalls are in its own file. Now you can diff them, grep specific workers, and correlate with your application logs by PID.

-k: stack trace per syscall

When you know a syscall is happening but can’t find where in the application it’s being called from:

strace -k -e trace=openat ./myapp 2>&1 | head -80

Each matching syscall prints the userspace call stack. Requires frame pointers in the binary (-fno-omit-frame-pointer) or debug symbols. Most distro packages strip them, so either rebuild or run against a debug build. On glibc-based systems you’ll at minimum get the library frames; the application frames depend on build flags.

Not something you’ll use every day. But when you have a mystery openat with no obvious caller — and adding logging to the binary isn’t an option — this is the move.

What you’re giving up

strace uses ptrace(2). Same mechanism as a debugger, same constraints: you need CAP_SYS_PTRACE or root, the traced process must be yours or a descendant, and you’re buying 2x–10x slowdown on syscall-intensive workloads. If your application makes 100,000 syscalls per second and you attach strace, you’ll double latency at minimum. This is the reason strace is a diagnostic tool and not a monitoring tool.

/proc/sys/kernel/perf_event_paranoid can limit what ptrace does in a hardened environment. If you get permission errors attaching to a process you own, check that setting and your container’s seccomp profile — most container runtimes block ptrace by default and that’s probably correct.

For lower-overhead syscall tracing where you can’t afford the ptrace penalty: bpftrace attaches via eBPF and has near-zero cost on non-matching syscalls, or dtrace covers the same ground on FreeBSD with a more mature set of probes. But both require setup and privileges and knowing the probe names. strace is already installed. -c -e -P -T and you have answers in 90 seconds.

The canonical reference for ptrace(2), setns(2), and what strace is actually doing under the hood is Kerrisk’s The Linux Programming Interface. The chapter on tracing is thorough and covers the kernel internals that explain why the performance hit exists and how the call intercept mechanism works. Reading it once will make you a better debugger and a worse passenger when someone else is doing debugging badly.