perf_event_open(2): hardware counters without the wrapper

perf stat ./myapp is convenient. It is also a shell around perf_event_open(2). If you want hardware counters inline, without spawning a subprocess, without writing a perf.data file, without the perf binary installed, call the syscall yourself.

The struct

#include <linux/perf_event.h>
#include <sys/ioctl.h>
#include <sys/syscall.h>
#include <unistd.h>

static inline int
perf_event_open(struct perf_event_attr *attr, pid_t pid, int cpu,
                int group_fd, unsigned long flags)
{
    return (int)syscall(SYS_perf_event_open, attr, pid, cpu, group_fd, flags);
}

No glibc wrapper. syscall(2) it. The struct perf_event_attr is large but safe to zero-initialize and fill in only what you need:

struct perf_event_attr attr = {
    .type           = PERF_TYPE_HARDWARE,
    .size           = sizeof(struct perf_event_attr),
    .config         = PERF_COUNT_HW_INSTRUCTIONS,
    .disabled       = 1,
    .exclude_kernel = 1,
    .exclude_hv     = 1,
};

.disabled = 1 means the counter does not run when the fd opens. Start it explicitly with ioctl. .exclude_kernel keeps the count user-space-only; without it you also count instructions executed on your behalf inside syscalls.

Event types

PERF_TYPE_HARDWARE with a PERF_COUNT_HW_* config taps the CPU’s performance monitoring unit:

PERF_COUNT_HW_CPU_CYCLES               raw CPU cycles
PERF_COUNT_HW_INSTRUCTIONS             retired instructions
PERF_COUNT_HW_CACHE_REFERENCES         last-level cache accesses
PERF_COUNT_HW_CACHE_MISSES             last-level cache misses
PERF_COUNT_HW_BRANCH_INSTRUCTIONS      branch instructions
PERF_COUNT_HW_BRANCH_MISSES            mispredicted branches
PERF_COUNT_HW_STALLED_CYCLES_FRONTEND  front-end pipeline stalls
PERF_COUNT_HW_STALLED_CYCLES_BACKEND   back-end pipeline stalls

PERF_TYPE_SOFTWARE uses kernel counters, not PMU hardware. No capability needed, available anywhere:

PERF_COUNT_SW_CPU_CLOCK         nanoseconds of CPU time (CLOCK_MONOTONIC-based)
PERF_COUNT_SW_TASK_CLOCK        nanoseconds this task was scheduled
PERF_COUNT_SW_PAGE_FAULTS       minor + major faults
PERF_COUNT_SW_CONTEXT_SWITCHES  voluntary and involuntary
PERF_COUNT_SW_CPU_MIGRATIONS    migrations across CPUs
PERF_COUNT_SW_PAGE_FAULTS_MAJ   major (disk-backed) faults only

PERF_TYPE_RAW with a vendor PMU event code reaches counters that the kernel has no named constant for. Intel and AMD publish these in their architecture manuals. Useful for LLC miss latency breakdowns, DTLB stalls, memory bandwidth utilization: the specifics that the generic constants don’t expose.

Targeting

The pid and cpu arguments control scope:

pid =  0, cpu = -1   current process, all CPUs         (the usual case)
pid =  N, cpu = -1   process N, all CPUs               (attach to running pid)
pid = -1, cpu =  N   all processes on CPU N            (needs CAP_PERFMON)
pid = -1, cpu = -1   invalid, returns EINVAL

System-wide (pid = -1) requires CAP_PERFMON, added in Linux 5.8 to replace the blunter CAP_SYS_ADMIN for this purpose. The sysctl /proc/sys/kernel/perf_event_paranoid also gates what unprivileged processes can do: values >= 2 restrict you to your own processes only. See the perf stat tip for the full paranoid table.

Reading a single counter

int fd = perf_event_open(&attr, 0, -1, -1, PERF_FLAG_FD_CLOEXEC);
if (fd < 0) err(1, "perf_event_open");

ioctl(fd, PERF_EVENT_IOC_RESET,   0);
ioctl(fd, PERF_EVENT_IOC_ENABLE,  0);

/* ... the work you're measuring ... */

ioctl(fd, PERF_EVENT_IOC_DISABLE, 0);

uint64_t count;
if (read(fd, &count, sizeof(count)) != sizeof(count))
    err(1, "read");
printf("instructions: %llu\n", (unsigned long long)count);
close(fd);

PERF_FLAG_FD_CLOEXEC closes the fd on exec. Leave it out and the counter fd leaks into every child process. This is not hypothetical; it bites you the first time you fork a subprocess inside a measurement region and wonder why the counts are wrong.

Grouping for atomic reads

Two separate read(2) calls have a race: the scheduler can preempt you between them. If you want cycles and instructions at the same instant, group them. The group leader controls scheduling; the kernel reads all members at once:

/* leader: cycles */
struct perf_event_attr a_cycles = {
    .type           = PERF_TYPE_HARDWARE,
    .size           = sizeof(a_cycles),
    .config         = PERF_COUNT_HW_CPU_CYCLES,
    .disabled       = 1,
    .exclude_kernel = 1,
    .read_format    = PERF_FORMAT_GROUP
                    | PERF_FORMAT_TOTAL_TIME_ENABLED
                    | PERF_FORMAT_TOTAL_TIME_RUNNING,
};
int leader = perf_event_open(&a_cycles, 0, -1, -1, PERF_FLAG_FD_CLOEXEC);
if (leader < 0) err(1, "perf_event_open leader");

/* follower: instructions, group_fd = leader */
struct perf_event_attr a_insns = {
    .type           = PERF_TYPE_HARDWARE,
    .size           = sizeof(a_insns),
    .config         = PERF_COUNT_HW_INSTRUCTIONS,
    .exclude_kernel = 1,
    .read_format    = PERF_FORMAT_GROUP
                    | PERF_FORMAT_TOTAL_TIME_ENABLED
                    | PERF_FORMAT_TOTAL_TIME_RUNNING,
};
int follower = perf_event_open(&a_insns, 0, -1, leader, PERF_FLAG_FD_CLOEXEC);
if (follower < 0) err(1, "perf_event_open follower");

Enable and disable the whole group via the leader fd:

ioctl(leader, PERF_EVENT_IOC_RESET,   PERF_IOC_FLAG_GROUP);
ioctl(leader, PERF_EVENT_IOC_ENABLE,  PERF_IOC_FLAG_GROUP);

/* ... work ... */

ioctl(leader, PERF_EVENT_IOC_DISABLE, PERF_IOC_FLAG_GROUP);

Read from the leader with a struct that matches PERF_FORMAT_GROUP:

struct {
    uint64_t nr;           /* number of group members (2 here) */
    uint64_t time_enabled;
    uint64_t time_running;
    uint64_t values[2];    /* one per member, in open order */
} result;

if (read(leader, &result, sizeof(result)) != sizeof(result))
    err(1, "read group");

uint64_t cycles = result.values[0];
uint64_t insns  = result.values[1];

That read is atomic from the kernel’s perspective: both counts come from the same scheduling window.

The multiplexing problem

Modern Intel CPUs have four generic PMU counter registers. AMD Zen 4 has six. Ask for more hardware events than there are physical registers and the kernel multiplexes: it time-shares the registers across your logical events, so each event runs only during its scheduled window.

PERF_FORMAT_TOTAL_TIME_ENABLED and PERF_FORMAT_TOTAL_TIME_RUNNING expose the ratio. When they differ, the raw count is partial:

if (result.time_running < result.time_enabled) {
    double scale = (double)result.time_enabled / result.time_running;
    cycles = (uint64_t)((double)cycles * scale);
    insns  = (uint64_t)((double)insns  * scale);
}

If time_running / time_enabled is below 0.90, the scaled count is shaky. The kernel is rotating too many events through too few registers and the error compounds. Either reduce the event set or measure in multiple passes. perf stat does this scaling silently and prints estimated values without telling you the ratio. At the syscall level you see the ratio and can decide whether the number is trustworthy.

A group of counters scheduled together counts as one PMU slot for multiplexing purposes. That is another reason to use groups: the members of a group stay correlated even under multiplexing. Separate fds on separate groups can rotate out of phase and produce useless IPC ratios.

FreeBSD

FreeBSD has hwpmc(4) and the pmc(3) / pmcstat(8) API. Allocate PMC resources via pmc_allocate(3), attach to a process, read via pmc_read(3). No group-atomic read equivalent: you sample each counter independently. For ad hoc measurement, pmcstat -S instructions -p $pid is the closest equivalent to perf stat -e instructions.

There is no portable syscall-level interface across both OSes. libpfm4 provides a cross-platform PMU event encoding layer but the plumbing beneath is still OS-specific.