$RodHat_/search
Console Tips

rctl(8) on FreeBSD: per-jail resource limits that actually deny

Published by

This article may contain Amazon affiliate links. I may earn a commission if you buy through them. Disclosure.

rctl(8) on FreeBSD: per-jail resource limits that actually deny
Photo: David Pupăză / Unsplash
On this page · 5 sections
  1. Turn accounting on
  2. Rule syntax
  3. Read what it is actually using
  4. Why not just login.conf
  5. The cons

A jail without limits is a polite suggestion. It isolates the filesystem, the process table and the network stack, then lets one runaway find inside it eat every page of RAM on the host. I have watched a single leaking daemon in a jail take down the other nine tenants and the box’s ssh daemon with it.

FreeBSD has had the answer since 9.0: rctl(8), the front end to the racct resource accounting framework. It is not on by default, and the reason people never find it is that without one tunable the commands just fail.

Turn accounting on

racct is a boot-time tunable, so it goes in /boot/loader.conf and needs a reboot:

echo 'kern.racct.enable=1' >> /boot/loader.conf
reboot

After that, rctl -u jail:web and friends work. Before it, you get an error about resource accounting not being enabled. Accounting has a small cost, since the kernel now counts things per process, per user, per login class and per jail. On anything hosting other people’s workloads I consider it cheap.

Rule syntax

A rule is subject:subject-id:resource:action=amount/per.

rctl -a jail:web:memoryuse:deny=2G
rctl -a jail:web:maxproc:deny=500
rctl -a jail:web:openfiles:deny=20000
rctl -a jail:web:writebps:throttle=50M
  • subject is process, user, loginclass or jail. Jails are the interesting one.
  • resource includes cputime, datasize, memoryuse, vmemoryuse, maxproc, openfiles, pseudoterminals, swapuse, nthr, pcpu, readbps, writebps, readiops, writeiops and wallclock. rctl(8) has the full list for your release; read it, it moves.
  • action is deny (the allocation fails), log (syslog it), devctl (send an event to devd), throttle (slow it down, I/O and CPU style resources only), or sig* such as sigterm and sigkill.
  • per says what the limit is scoped to. jail:web:maxproc:deny=500/user is 500 per user inside the jail, not 500 total.

The difference between deny and a signal is the whole design decision. deny=2G on memoryuse is not an OOM kill. Allocations past the limit fail, and whether the daemon survives depends on whether it checks malloc() returns, which, well. sigkill is blunter and honest. For anything with a supervisor to restart it, I use log first to learn the real numbers and deny only after.

Read what it is actually using

rctl -u jail:web

That prints the current usage for every resource on the jail, so you can set limits from measurement rather than a guess. -h makes the byte counts human-readable. List the rules you have installed with rctl -l jail:web, or rctl -l for everything, and remove one with rctl -r jail:web:maxproc.

If you are setting rules by hand on a running system, they vanish at reboot. Persist them in /etc/rctl.conf, one rule per line in the same syntax, and set rctl_enable="YES" in rc.conf. Do not paste the rctl -a lines into a startup script and call it done; the rules should load before the jails start, which is what the rc ordering does for you.

Why not just login.conf

login.conf limits (datasize, maxproc, and so on) are per-process or per-login-class rlimits, applied when a session starts. They are fine for users. They do not sum across a jail, so a jail with 400 processes each under its rlimit can still hold 400 times the memory you meant to allow. rctl accounts at the jail level, which is the number you actually care about. I covered building the jail itself in ZFS and jails as poor man’s containers; this is the missing half of that setup.

For the Linux equivalent, the interface is cgroups, and the cgroups v2 without systemd post walks through memory.max and pids.max, which are the same two knobs by different names. Rod will say the BSD syntax is one line and the cgroup one is a directory tree of files. Rod is right.

The cons

  • Needs a reboot to enable, so you plan it, you do not discover it during an incident.
  • deny on memory does not kill anything. Applications that ignore allocation failure will misbehave in ways more confusing than a clean OOM kill.
  • memoryuse counts resident memory. It does not make the ZFS ARC go away; ARC is kernel memory and is not charged to the jail, so cap it separately as in the ARC is not a leak.
  • Rules are by jail name. Rename the jail and the rule is orphaned, silently.
  • Do not set a limit you have not measured. A maxproc that is too low produces fork failures that look like application bugs for an afternoon.

Start with log actions on a staging jail for a week, read rctl -u against your peak, then flip them to deny with margin. The first time a leaking worker hits its ceiling and the other tenants never notice, you will stop thinking of this as optional.