Topic
ZFS & Storage
Practical ZFS, filesystems, backups, replication, and storage failure analysis for operators.
cgroups v2 is just files. Here's how to use them without touching systemd.
Linux cgroups v2 exposes memory, CPU, and I/O limits through a plain filesystem interface under /sys/fs/cgroup/. No systemd, no container runtime — write to files, read the results. Here's the real workflow for memory.max, cpu.max, freezer, and io.max.
OpenZFS 3.0 ships the RAIDZ expansion and the dedup engine they finally got right
OpenZFS 3.0 landed with RAIDZ expansion, a ground-up rewrite of the dedup engine, and block cloning. Three features that have been "coming soon" for most of the past decade are now in a release build. Worth looking at.
bpftrace gives Linux what FreeBSD had in 2005. The one-liners are worth the wait.
dtrace showed up on Solaris in 2004, shipped in FreeBSD 7 in 2008, and Linux users spent the next decade pretending strace was sufficient. bpftrace is the real answer — dynamic kernel tracing, histograms, stack walks, zero overhead when idle.
The storage was fine. We were swapping.
Three weeks, two engineers, one open storage vendor ticket, and roughly forty collective hours chasing read latency on a ZFS pool. The pool was fine the whole time. There was a swapfile nobody remembered adding seven months earlier.
pledge() turns 10. Linux still doesn't have anything half as clean.
OpenBSD's pledge(2) landed in 5.9 in 2016. Ten years later, Linux has Landlock (good) and seccomp-bpf (powerful and painful) and still nothing that lets a process sandbox itself in a single readable line. RodHat on why API simplicity is a security property.
Upgrade like you can undo it, because with bectl you can
A ZFS boot environment is a bootable clone of your root dataset. Make one before every upgrade and a wrecked kernel becomes a reboot, not a recovery-media evening. The bectl walk, and the loader trick that saves you when the new one won't boot.
Your backups are fine and one account suspension deletes all of them
Three copies, two media, one offsite — and all three sitting under a single cloud identity that a fraud algorithm can disable at 4am with no appeal path. The rule was written when a copy meant a physical object somebody had to come and take.
Your shell pipeline succeeded because the last command was polite
A failed producer can disappear behind a successful consumer. RodHat explains pipeline exit status, pipefail, PIPESTATUS, and how to stop backup scripts from lying.
zfs send is the backup tool you already have and probably aren't using
An incremental zfs send moves exactly the blocks that changed, verified end to end, with no file-tree walk at all. The full replication pipeline is one command, and the resume token turns a failed 3TB transfer from a restart into a continuation.
The temporary NFS mount that ran production for six years
Somebody stood it up in an afternoon to unblock a launch. It had no monitoring, no owner, no backup, and no entry in any diagram. It survived two datacentre moves, a company acquisition, and every engineer who knew it existed.
The six rsync flags worth knowing, and the one that builds you free snapshots
-a doesn't preserve what you think it preserves. --inplace and --partial solve opposite problems and combining them wrong loses data. And --link-dest turns rsync into a deduplicating snapshot system in one flag.
Stop your backup job from taking production down with it
nice does nothing for I/O. ionice only works on one scheduler that most systems no longer use. cgroup v2 io.latency and io.max are the two knobs that actually work, and they work on the thing you already have.
Get a real disk latency histogram in one line of bpftrace
iostat gives you an average, and averages hide the tail that's actually hurting you. A four-line bpftrace program prints a log2 histogram of block I/O latency per device, live, on a production box, with no agent and no restart.
ZFS boot environments + jails will do 90% of what you're using Docker for, and the other 10% is the part you don't actually need
A real walkthrough of FreeBSD jails on ZFS datasets as a lighter, more honest isolation model than a container runtime for a lot of workloads.
No, ZFS did not eat your RAM. Read arc_summary before you tune anything.
The ARC is designed to consume all the memory you aren't using and give it back under pressure. But it does give it back slowly, and there are two real cases where you should cap it. Here's how to tell which situation you're actually in.
The restore worked perfectly and we still lost eleven days
We tested our backups. Monthly, documented, signed off. The restore ran clean, the checksums matched, the database came up on the first try. And the data in it was eleven days old, because for eleven days the job had been backing up a directory nobody was writing to any more.
The disk is full and du says it isn't. It's an open file descriptor.
df reports 100%, du walks the whole tree and finds nothing. The space is in a file somebody deleted while a process still had it open — the inode lives until the last fd closes. lsof +L1 finds it in one command.