$RodHat_/search
Console Tips

Pin processes, threads and interrupts to CPUs with FreeBSD cpuset(1)

Published by

This article may contain Amazon affiliate links. I may earn a commission if you buy through them. Disclosure.

Terminal showing cpuset -g output listing the CPU mask of a process on FreeBSD
AI illustration: RodHat
On this page · 6 sections
  1. Read before you write
  2. Pin a new command or a running one
  3. Move an interrupt off the busy cores
  4. Confine a jail
  5. Prove it did something
  6. Gotchas

Linux people reach for taskset. FreeBSD has had cpuset(1) since 7.1, it does more, and almost nobody on the BSD side uses it until a database and a NIC interrupt start fighting over core 0 at two in the morning.

Read before you write

cpuset -g
cpuset -g -p 1234

-g gets. With no target it reports the current process. With -p it reports that pid. You see two lines, the set id and the mask, something like pid 1234 mask: 0, 1, 2, 3. Always look first. A jailed or rctl-limited process may already have a narrower mask than you expect, and you cannot widen past what the parent set allows.

Pin a new command or a running one

cpuset -l 2-3 ./worker
cpuset -l 2-3 -p 1234

-l is the CPU list, ranges and commas both fine. The first form starts the command inside a new set. The second moves a running process. Both change the whole process, every thread. To touch one thread, use -t with a thread id, which you can get from procstat -t <pid>. I covered procstat in reading a wedged process; same tool, different question.

The Linux equivalent, plus NUMA policy, is in taskset and numactl. The semantics differ in one way worth knowing: cpusets are hierarchical. A child set is always a subset of its parent. Pinning is a restriction inside a tree, not a free-form mask, which is why a pin that tries to escape the jail’s set simply fails.

Move an interrupt off the busy cores

This is the part taskset cannot do:

vmstat -i
cpuset -g -x 257
cpuset -l 1 -x 257

vmstat -i lists interrupt sources with their numbers and rates. -x takes the IRQ number. Find the NIC queue interrupt that is burning cycles, read where it lands, and bind it to a core that is not running your application’s hot threads. For a multiqueue NIC you want queues spread across cores, and the application’s network threads on the same cores as their queues, not on whatever the scheduler felt like. Whether a given driver honours the binding, and how many queues it exposes, depends on the driver. Check its man page and vmstat -i after, rather than trusting that the command returned zero.

Confine a jail

cpuset -l 4-7 -j 3

-j takes a jail id from jls. Every process in that jail, current and future, is held to those CPUs. For a noisy tenant that is a blunt and effective cap on CPU parallelism. It does not limit CPU time within those cores, so pair it with rctl if you also want a percentage ceiling. Cores restrict where, rctl restricts how much. Different problems.

Prove it did something

Pinning is superstition without measurement. Run your benchmark before and after and feed both sets of samples to ministat. If the difference is inside the noise, you added a constraint for nothing, and the scheduler was already doing fine. top -P shows per-CPU load, so you can at least see whether the pinned cores are the ones working.

Gotchas

  • A mask is not a reservation. Other processes still run on your pinned cores unless you also move them off.
  • Pin a multithreaded program to one core and you have built a very elaborate single-threaded program.
  • On a box with several NUMA domains, a flat CPU list ignores memory locality. cpuset(1) has a domain option for that on current releases. Read the man page for your version, since I am not going to quote flags from memory for a feature that moved around.
  • Nothing here survives a reboot. Put the command in the rc script that starts the service, not in your shell history.

Old trick, same as ever: the scheduler is usually smarter than you. It stops being smarter than you exactly when you know something it does not, such as which interrupt belongs with which thread. More in the FreeBSD and BSD hub.