$RodHat_/search
Console Tips

PSI: /proc/pressure tells you a box is dying before load average does

Published by · Updated

This article may contain Amazon affiliate links. I may earn a commission if you buy through them. Disclosure.

Analog pressure gauge with the needle near 8 bar
Photo: Amir Ghoorchiani / Pexels
On this page · 4 sections
  1. What “some” and “full” mean
  2. Triggers: poll() instead of polling
  3. Per-cgroup pressure
  4. Where it bites

Load average is a number from 1993 that mixes runnable tasks with tasks stuck in uninterruptible sleep, then smooths it over one, five and fifteen minutes. A load of 40 might be a CPU melting, a dead NFS mount, or one process hammering a slow disk. The number cannot tell you which. I have watched grown adults page each other over it.

PSI answers the question load average was always being asked. Since Linux 4.20 the kernel tracks how much time tasks spent waiting on a resource, and exposes it in three files:

$ cat /proc/pressure/cpu /proc/pressure/memory /proc/pressure/io
some avg10=0.00 avg60=0.00 avg300=0.00 total=104827339
some avg10=0.31 avg60=0.12 avg300=0.04 total=9114532
full avg10=0.18 avg60=0.07 avg300=0.02 total=6120047
some avg10=2.40 avg60=1.10 avg300=0.35 total=88123411
full avg10=1.95 avg60=0.88 avg300=0.28 total=71200988

The averages are percentages of wall-clock time over 10, 60 and 300 seconds. total is cumulative stall in microseconds, which is the field you want if you are writing a monitor, because you can diff it yourself and not depend on the kernel’s smoothing.

What “some” and “full” mean

some: at least one runnable task was stalled on the resource. The CPU was possibly busy with something else, but somebody waited.

full: every non-idle task was stalled at the same time. Nothing useful got done. For memory and IO this is the number that matters. A full of a few percent on memory means the box is spending real time in reclaim or swap with no productive work happening, and it has been doing so for seconds.

The CPU file has no meaningful full at the system level; the line is there for format consistency (the docs say it is zero there, and the cgroup version is where it gets a value, as of 5.13). Do not alert on it.

Triggers: poll() instead of polling

The averages are fine for a dashboard. For reaction, write a threshold into the file and wait for the kernel to wake you:

import os, select

fd = os.open("/proc/pressure/memory", os.O_RDWR | os.O_NONBLOCK)
os.write(fd, b"some 150000 1000000\0")   # 150ms stall within any 1s window

p = select.poll()
p.register(fd, select.POLLPRI)
while True:
    for _, ev in p.poll():
        if ev & select.POLLERR:
            raise SystemExit("trigger died")
        if ev & select.POLLPRI:
            print("memory stalled >=150ms in the last 1s. shed load now.")

Format is <some|full> <stall threshold µs> <window µs>. The kernel docs set the window between 500 ms and 10 s. Note the trailing NUL; the example in the kernel docs writes strlen(trig) + 1 bytes and so should you. The trigger lives as long as the fd stays open, and events are rate limited to one per window, so a sustained stall does not flood you.

This is how userspace OOM daemons (earlyoom-style tools, lmkd on Android) decide to kill something before the kernel’s OOM killer shows up late and picks the wrong victim. If you have read the OOM killer was doing its job, a trigger on memory is the thing that would have fired first.

Per-cgroup pressure

On cgroup v2 every group gets cpu.pressure, memory.pressure and io.pressure, same format, same trigger syntax. That is the useful granularity: the host can look fine while one container sits at 40% full memory pressure inside its limit. If you are building those groups by hand, see cgroups v2 without systemd, and for why RSS alone will not explain what you find there, PSS accounting.

Where it bites

  • It can be compiled in but off. Kernels built with CONFIG_PSI_DEFAULT_DISABLED need psi=1 on the command line. If /proc/pressure/ is missing or reads fail, check that before blaming yourself.
  • It measures stall, not cause. High IO pressure tells you tasks are waiting on IO. It does not tell you whose IO. Follow it with bpftrace or perf, not a guess.
  • Averages hide bursts. avg10=0.5 can be one 50 ms stall or a hundred tiny ones. Use total deltas or a trigger when burst shape matters.
  • Triggers need care with privilege. Opening and writing the system-wide files is root business in practice, and the docs add restrictions on small windows for unprivileged users. Test as the user your daemon will run as.

I rolled my eyes at PSI when it landed. Another accounting layer, another file to forget about. Then I diffed total on a wedged database host and saw full memory stall climb a second at a time while load average sat there looking calm. Fine. It’s good. Put a trigger on it.

Sources

  1. PSI - Pressure Stall Information (The Linux Kernel documentation)
  2. Control Group v2 (PSI interface files) (The Linux Kernel documentation)