$RodHat_

Topic · 24 articles

ZFS & Storage

ZFS is the best thing to happen to storage in thirty years and most people use maybe a fifth of it. Replication, rollback, and honest memory behaviour are already on the box; the work is learning which command does what before the disk fills up at 3am.

Start with replication and boot environments, because those are the two that save your job. Then the ARC, so you stop 'fixing' memory that isn't broken. The Tales are a catalogue of every way a disk can be full while claiming it isn't, and every way a backup can succeed while saving nothing.

Start here

Read these first, in this order.

  1. 01

    Tip · 2026-07-31

    zfs send is the backup tool you already have and probably aren't using

    An incremental zfs send moves exactly the blocks that changed, verified end to end, with no file-tree walk at all. The full replication pipeline is one command, and the resume token turns a failed 3TB transfer from a restart into a continuation.

  2. 02

    Tip · 2026-08-07

    Upgrade like you can undo it, because with bectl you can

    A ZFS boot environment is a bootable clone of your root dataset. Make one before every upgrade and a wrecked kernel becomes a reboot, not a recovery-media evening. The bectl walk, and the loader trick that saves you when the new one won't boot.

  3. 03

    Tip · 2026-07-21

    No, ZFS did not eat your RAM. Read arc_summary before you tune anything.

    The ARC is designed to consume all the memory you aren't using and give it back under pressure. But it does give it back slowly, and there are two real cases where you should cap it. Here's how to tell which situation you're actually in.

  4. 04

    Tip · 2026-07-30

    The six rsync flags worth knowing, and the one that builds you free snapshots

    -a doesn't preserve what you think it preserves. --inplace and --partial solve opposite problems and combining them wrong loses data. And --link-dest turns rsync into a deduplicating snapshot system in one flag.

  5. 05

    Tip · 2026-07-19

    The disk is full and du says it isn't. It's an open file descriptor.

    df reports 100%, du walks the whole tree and finds nothing. The space is in a file somebody deleted while a process still had it open, and the inode lives until the last fd closes. lsof +L1 finds it in one command.

The syscalls underneath

Version introduced, sourced from the man page. Full cards in the syscall reference, or check your kernel.

InterfaceSinceWhat it does
copy_file_range(2)Linux 4.5Copy a range of bytes between two file descriptors entirely inside the kernel, with server-side offload on NFS 4.2 and reflink on btrfs.
statx(2)Linux 4.11Extended stat: request specific fields, get birth time, inode attribute flags and mount ID.
fsopen(2)Linux 5.2Open a filesystem context: step one of the fd-based mount API (fsconfig, fsmount, move_mount).
flock(2)4.2BSD; Linux 2.0Apply or remove an advisory whole-file lock tied to an open file description.

More console tips (6)

Rod's tales (9)

MOTD (4)

Other topics