$RodHat_
Console Tips

cgroups v2 is just files. Here's how to use them without touching systemd.

Published by

cgroups v2 is just files. Here's how to use them without touching systemd.
Photo: AI-generated — no human photographer / RodHat AI Cover

The original cgroups interface was a mess. Not a “gets the job done but I wouldn’t put it in a textbook” mess — a genuine architectural embarrassment. You had a separate hierarchy per controller, so memory limits lived under /sys/fs/cgroup/memory/, CPU limits under /sys/fs/cgroup/cpu/cpuacct/, and you had to separately manage process membership in each tree. Nobody ever had all the limits consistently set on the processes that actually mattered. Containers were invented partly to paper over this.

cgroups v2 is better. A single unified hierarchy under /sys/fs/cgroup/. One directory per resource group, one cgroup.procs file, all controllers in the same place. It’s been the default since Linux 5.10 (2020), which means if your kernel is less than six years old, you have it. The unified design is genuinely cleaner and I’ll say it plainly.

The problem is what happened next: systemd ate the interface. On any systemd-based distro, PID 1 claims the cgroupv2 root at boot. The kernel only allows a single manager to own the unified hierarchy — so if systemd is there, you work within its slice structure or you fight it. On a non-systemd system (Alpine with OpenRC, FreeBSD running Linux binaries, a minimal VM with s6 or runit), the raw interface is entirely yours.

This tip covers the raw interface. On systemd, use systemd-run --scope --slice=batchjobs.slice to create a transient scope instead of creating cgroups manually — the mechanics of limits are the same, the path to get there differs.

Verify you’re on v2

mount | grep cgroup
# cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)

One mount, type cgroup2, unified hierarchy. If you see multiple mounts for different controllers (cgroup type, not cgroup2), you’re on v1 and the rest of this won’t apply.

Available controllers on this kernel:

cat /sys/fs/cgroup/cgroup.controllers
# cpuset cpu io memory hugetlb pids rdma misc

cat /sys/fs/cgroup/cgroup.subtree_control
# cpuset cpu io memory pids

cgroup.controllers is what the kernel has compiled in. cgroup.subtree_control is what’s enabled for child cgroups. If a controller is in the first but not the second, children can’t use it. Enable what you need:

echo '+memory +cpu +io' > /sys/fs/cgroup/cgroup.subtree_control

On a systemd machine this write will fail with EACCES — systemd owns the root cgroup. On a bare system, it works.

Creating a cgroup

mkdir /sys/fs/cgroup/batchjobs

That’s it. The kernel populates the directory with all control files immediately:

ls /sys/fs/cgroup/batchjobs/
# cgroup.controllers  cgroup.events  cgroup.freeze  cgroup.kill  cgroup.procs
# cgroup.subtree_control  cgroup.type  cpu.max  cpu.weight  io.max  io.stat
# memory.current  memory.events  memory.high  memory.max  memory.peak
# memory.stat  pids.current  pids.max  ...

Enable the controllers you want available within this cgroup:

echo '+memory +cpu +io' > /sys/fs/cgroup/batchjobs/cgroup.subtree_control

Adding processes

Write a PID to cgroup.procs:

echo $$ > /sys/fs/cgroup/batchjobs/cgroup.procs

The current shell and everything it forks is now in batchjobs. A process can only belong to one leaf cgroup — writing it to a new one removes it from the old one. Children inherit their parent’s cgroup at fork(), so putting a parent in the cgroup catches its entire process tree.

Verify:

cat /sys/fs/cgroup/batchjobs/cgroup.procs
# 8294
# 8297

Memory limits

Two knobs, two behaviors:

# Soft cap — kernel reclaims aggressively when usage exceeds this, but won't kill
echo '256M' > /sys/fs/cgroup/batchjobs/memory.high

# Hard cap — OOM killer fires within the cgroup when this is hit
echo '512M' > /sys/fs/cgroup/batchjobs/memory.max

memory.high slows allocations and triggers reclaim when exceeded. memory.max kills. Set memory.high well below memory.max to get a graceful pressure phase before the kill. For a batch job you want to contain but not murder, memory.high alone is often the right tool.

Before setting either, know what the process actually uses. The RSS from ps or top is misleading — it includes shared pages counted against every process that maps them. Proportional Set Size (PSS) is accurate:

grep '^Pss:' /proc/$PID/smaps_rollup
# Pss:              183412 kB

That’s ~179 MiB actual private + proportional-shared footprint. See the smaps and PSS tip for the full accounting breakdown. Set memory.max at 1.5–2× the measured PSS with room for working-set growth. Set it tighter and you’ll get sporadic OOM kills under load that look like bugs.

Check current and peak usage:

cat /sys/fs/cgroup/batchjobs/memory.current   # bytes, live
cat /sys/fs/cgroup/batchjobs/memory.peak      # high-water mark since cgroup creation

Check whether limits were hit:

cat /sys/fs/cgroup/batchjobs/memory.events
# low 0
# high 14
# max 0
# oom 0
# oom_kill 0

high 14 means the soft limit was crossed fourteen times. oom_kill 0 means nothing has been killed. This is how you tune limits against a live workload before committing: run the process, watch memory.events, raise or lower memory.high until high stays near zero under normal operation. The cgroup OOM killer is a better alternative to depending on the system-wide OOM killer firing somewhere deep in the machine and killing the wrong thing — the tale about the OOM killer doing its job covers exactly that failure mode.

CPU limits

Two knobs with different semantics.

cpu.weight is a relative scheduling priority, 1–10000, default 100:

echo '50' > /sys/fs/cgroup/batchjobs/cpu.weight    # half normal priority
echo '200' > /sys/fs/cgroup/batchjobs/cpu.weight   # double normal priority

Weight-based scheduling only applies when the system is contested. If everything else is idle, a low-weight cgroup can still use 100% CPU. This is usually what you want for batch workloads: fast when the machine is free, throttled when interactive traffic shows up.

cpu.max is a hard bandwidth quota. Format: $quota $period in microseconds, or max $period for unlimited:

# 25% of one CPU core: 250ms consumed per 1000ms window
echo '250000 1000000' > /sys/fs/cgroup/batchjobs/cpu.max

# Exactly one CPU core's worth, regardless of how many cores the box has
echo '1000000 1000000' > /sys/fs/cgroup/batchjobs/cpu.max

# Remove the quota
echo 'max 1000000' > /sys/fs/cgroup/batchjobs/cpu.max

Hard quotas apply globally — the cgroup can’t burst above the cap even on an otherwise idle machine. Use cpu.weight for priority-based coexistence. Reach for cpu.max when you need an actual ceiling: billing isolation, containing a process that will use everything available, pinning a workload to one core’s equivalent on a NUMA box. For CPU affinity (which CPUs, not how much CPU), see taskset and numactl — the two knobs are orthogonal.

Check actual CPU usage:

cat /sys/fs/cgroup/batchjobs/cpu.stat
# usage_usec 483291042
# user_usec 412847103
# system_usec 70443939
# nr_periods 483291
# nr_throttled 12483
# throttled_usec 8294183

nr_throttled is how many scheduling periods were throttled. throttled_usec is total wall time lost to throttling. If nr_throttled / nr_periods is consistently above ~5%, the quota is too tight and the workload is suffering.

I/O limits

io.max throttles bandwidth and IOPS per block device. You need the major:minor device number:

ls -l /dev/sda
# brw-rw---- 1 root disk 8, 0 Aug 23 07:14 /dev/sda
# → "8:0"
# Bandwidth limits
echo '8:0 rbps=52428800 wbps=20971520' > /sys/fs/cgroup/batchjobs/io.max
# (50 MiB/s reads, 20 MiB/s writes)

# IOPS limits
echo '8:0 riops=1000 wiops=500' >> /sys/fs/cgroup/batchjobs/io.max

# Both — the kernel merges them per device
echo '8:0 rbps=52428800 wbps=20971520 riops=1000 wiops=500' > /sys/fs/cgroup/batchjobs/io.max

Check actual I/O:

cat /sys/fs/cgroup/batchjobs/io.stat
# 8:0 rbytes=1073741824 wbytes=268435456 rios=8192 wios=2048 dbytes=0 dios=0

Note: io.max requires the block stack to use blk-mq (multi-queue I/O), which has been the default since kernel 5.0. Legacy single-queue drivers don’t support cgroupv2 I/O throttling. Check with lsblk -t — if SCHED shows a cfq or deadline scheduler, you’re on the old path.

Freezing a cgroup

Write 1 to cgroup.freeze. Every process in the cgroup stops mid-instruction — no signal, no cooperation required:

echo 1 > /sys/fs/cgroup/batchjobs/cgroup.freeze

cat /sys/fs/cgroup/batchjobs/cgroup.events
# populated 1
# frozen 1

# Thaw
echo 0 > /sys/fs/cgroup/batchjobs/cgroup.freeze

The frozen processes aren’t dead — they’re suspended. They can’t receive signals, can’t run, can’t allocate. This is how container runtimes implement pause/resume. You can use it from a shell to freeze a runaway batch job while a time-sensitive operation runs, then thaw it when you’re done. Cleaner than SIGSTOP because it applies recursively to the entire cgroup tree, including processes the frozen parent would have forked after the freeze.

Cleanup

A cgroup can only be removed if it has no processes and no child cgroups:

# Verify empty
cat /sys/fs/cgroup/batchjobs/cgroup.procs   # should be empty

rmdir /sys/fs/cgroup/batchjobs

If rmdir fails with EBUSY, something’s still in there. Nuclear option:

echo 1 > /sys/fs/cgroup/batchjobs/cgroup.kill
# SIGKILL to everything in the cgroup and all descendants
# Wait for processes to exit, then:
rmdir /sys/fs/cgroup/batchjobs

cgroup.kill is the one file that does what it sounds like.

The full kernel documentation is in Documentation/admin-guide/cgroup-v2.rst in the kernel source tree — it’s well-maintained and actually readable, which puts it in rare company. Kerrisk’s The Linux Programming Interface covers cgroup fundamentals in the context of the broader Linux process model, which helps if you’re coming to this from first principles. For the OS scheduling and resource isolation theory underneath all of this, Operating Systems: Three Easy Pieces is the right starting point and it’s free online. The practical tracing tools that tell you what a cgroup is doing in real time — ftrace for kernel call paths, bpftrace for per-event analysis — are covered in the ftrace tip and the bpftrace one-liners tip.