Load average is a number from 1993 that mixes runnable tasks with tasks stuck in uninterruptible sleep, then smooths it over one, five and fifteen minutes. A load of 40 might be a CPU melting, a dead NFS mount, or one process hammering a slow disk. The number cannot tell you which. I have watched grown adults page each other over it.
PSI answers the question load average was always being asked. Since Linux 4.20 the kernel tracks how much time tasks spent waiting on a resource, and exposes it in three files:
$ cat /proc/pressure/cpu /proc/pressure/memory /proc/pressure/io
some avg10=0.00 avg60=0.00 avg300=0.00 total=104827339
some avg10=0.31 avg60=0.12 avg300=0.04 total=9114532
full avg10=0.18 avg60=0.07 avg300=0.02 total=6120047
some avg10=2.40 avg60=1.10 avg300=0.35 total=88123411
full avg10=1.95 avg60=0.88 avg300=0.28 total=71200988
The averages are percentages of wall-clock time over 10, 60 and 300 seconds.
total is cumulative stall in microseconds, which is the field you want if you
are writing a monitor, because you can diff it yourself and not depend on the
kernel’s smoothing.
What “some” and “full” mean
some: at least one runnable task was stalled on the resource. The CPU was
possibly busy with something else, but somebody waited.
full: every non-idle task was stalled at the same time. Nothing useful got
done. For memory and IO this is the number that matters. A full of a few
percent on memory means the box is spending real time in reclaim or swap with
no productive work happening, and it has been doing so for seconds.
The CPU file has no meaningful full at the system level; the line is there
for format consistency (the docs say it is zero there, and the cgroup version
is where it gets a value, as of 5.13). Do not alert on it.
Triggers: poll() instead of polling
The averages are fine for a dashboard. For reaction, write a threshold into the file and wait for the kernel to wake you:
import os, select
fd = os.open("/proc/pressure/memory", os.O_RDWR | os.O_NONBLOCK)
os.write(fd, b"some 150000 1000000\0") # 150ms stall within any 1s window
p = select.poll()
p.register(fd, select.POLLPRI)
while True:
for _, ev in p.poll():
if ev & select.POLLERR:
raise SystemExit("trigger died")
if ev & select.POLLPRI:
print("memory stalled >=150ms in the last 1s. shed load now.")
Format is <some|full> <stall threshold µs> <window µs>. The kernel docs set the
window between 500 ms and 10 s. Note the trailing NUL; the example in the kernel
docs writes strlen(trig) + 1 bytes and so should you. The trigger lives as
long as the fd stays open, and events are rate limited to one per window, so a
sustained stall does not flood you.
This is how userspace OOM daemons (earlyoom-style tools, lmkd on Android) decide
to kill something before the kernel’s OOM killer shows up late and picks the
wrong victim. If you have read the OOM killer was doing its job,
a trigger on memory is the thing that would have fired first.
Per-cgroup pressure
On cgroup v2 every group gets cpu.pressure, memory.pressure and
io.pressure, same format, same trigger syntax. That is the useful granularity:
the host can look fine while one container sits at 40% full memory pressure
inside its limit. If you are building those groups by hand, see
cgroups v2 without systemd, and
for why RSS alone will not explain what you find there,
PSS accounting.
Where it bites
- It can be compiled in but off. Kernels built with
CONFIG_PSI_DEFAULT_DISABLEDneedpsi=1on the command line. If/proc/pressure/is missing or reads fail, check that before blaming yourself. - It measures stall, not cause. High IO pressure tells you tasks are waiting
on IO. It does not tell you whose IO. Follow it with
bpftraceorperf, not a guess. - Averages hide bursts.
avg10=0.5can be one 50 ms stall or a hundred tiny ones. Usetotaldeltas or a trigger when burst shape matters. - Triggers need care with privilege. Opening and writing the system-wide files is root business in practice, and the docs add restrictions on small windows for unprivileged users. Test as the user your daemon will run as.
I rolled my eyes at PSI when it landed. Another accounting layer, another file
to forget about. Then I diffed total on a wedged database host and saw
full memory stall climb a second at a time while load average sat there
looking calm. Fine. It’s good. Put a trigger on it.
