Stop your backup job from taking production down with it

The backup starts at 2am. Between 2am and 2:40am, p99 on the API triples. Somebody adds nice -n 19 to the backup job and it changes nothing at all, because nice is a CPU scheduler hint and the backup is not competing for CPU. It’s competing for disk.

So they try ionice -c 3. That also does nothing, and for a more interesting reason.

Why ionice is probably a no-op on your box

ionice sets a priority for the CFQ I/O scheduler. CFQ was removed from Linux in 5.0. Modern kernels default to mq-deadline, bfq, or none (for NVMe, where the device does its own queuing).

Check:

cat /sys/block/nvme0n1/queue/scheduler
[none] mq-deadline kyber bfq

If that says none or mq-deadline, ionice class and priority are advisory to a scheduler that doesn’t read them. bfq does honor ionice — it’s CFQ’s successor — so on a bfq device ionice -c 3 (idle class) genuinely works. On everything else you’re shouting at a scheduler that isn’t listening.

That’s the whole reason this problem keeps getting “fixed” and staying broken.

What actually works: cgroup v2

Check you’re on unified hierarchy — nearly everything is by now:

mount | grep cgroup2
stat -fc %T /sys/fs/cgroup    # "cgroup2fs"

Two knobs, and they’re for different jobs.

io.max — a hard cap

mkdir /sys/fs/cgroup/backup
echo "259:0 rbps=52428800 wbps=52428800" > /sys/fs/cgroup/backup/io.max
echo $$ > /sys/fs/cgroup/backup/cgroup.procs
exec /usr/local/bin/nightly-backup

50MB/s read and write, absolute ceiling, always enforced. 259:0 is the device major:minor from lsblk. You can also set riops/wiops.

Simple and blunt. The downside is that it’s blunt: the backup is throttled to 50MB/s at 3pm when the array is idle and nobody would have noticed it running at 400MB/s. You’ve traded an incident for permanently slower backups.

io.latency — the one you actually want

Instead of capping the noisy job, you protect the important one:

mkdir /sys/fs/cgroup/database
echo "259:0 target=10" > /sys/fs/cgroup/database/io.latency
echo <postgres-pid> > /sys/fs/cgroup/database/cgroup.procs

That says: this cgroup’s I/O should complete within 10ms. The kernel watches. When the database’s observed latency starts exceeding its target, the kernel throttles every other cgroup on that device until it comes back under. When the database is idle, everyone else runs at full speed.

That’s the correct shape for this problem. You are not guessing a bandwidth number for the backup. You are stating a requirement for the thing that matters and letting the kernel do the arithmetic — and the backup gets the entire array at 4am when the database is quiet, which is exactly what you wanted all along.

Set the target from measurement, not vibes. Get the real distribution first — a bpftrace histogram takes four lines — and set the target somewhere above your current healthy p95. Set it below what the device can do and you’ll throttle everything permanently, which is a fun outage to diagnose.

Watch it work:

cat /sys/fs/cgroup/database/io.stat
cat /sys/fs/cgroup/backup/io.pressure

io.pressure is PSI — the fraction of wall time tasks in that cgroup spent stalled on I/O. If some avg10 on the backup cgroup climbs while the database’s stays flat, the mechanism is doing precisely its job.

The caveats that will bite you

Writeback. Buffered writes are flushed by kernel threads, and attributing them back to the cgroup that dirtied the page requires memory-cgroup accounting to be on and the filesystem to support it. It works on ext4 and btrfs. On ZFS it does not — ZFS has its own I/O pipeline and does not participate in cgroup I/O control at all. On ZFS your levers are zfs_vdev_*_max_active queue depths and dataset-level tuning, which is a different article and a worse experience.

io.latency needs a real queue. On NVMe with none as the scheduler and hardware that can do a million IOPS, contention may never build up enough for the throttle to engage, and your problem was never the block layer — it’s the filesystem, or fsync, or a lock.

Delegation. If systemd owns the hierarchy on your box, writing directly to /sys/fs/cgroup gets stomped on reconfiguration. Use the delegation mechanism it provides, or accept that your settings evaporate at the worst moment.

The summary

nice is CPU. ionice needs bfq. io.max is a hard cap that costs you throughput you didn’t need to give up. io.latency protects the workload you care about and lets everything else use whatever’s left — which is what everyone actually meant when they typed nice -n 19 and hoped.