OpenZFS 3.0 ships the RAIDZ expansion and the dedup engine they finally got right

ZFS dedup has been technically present in OpenZFS since before “OpenZFS” was even the name. It shipped in Solaris 10 in 2009. Seventeen years of it being available and seventeen years of experienced administrators telling you not to touch it unless your RAM budget is measured in units that rhyme with “terabytes.”

The dedup table — the DDT — lived in memory. Every unique block checksum got an entry. The entry stayed in the DDT for as long as the block existed. Your dedup ratio looked great in zpool status, your ARC looked fine in kstat, and then one morning you got a page because the kernel was evicting ARC entries to keep the DDT resident and your pool’s IOPS had cratered to the floor. The canonical advice was: if you’re asking whether you have enough RAM for ZFS dedup, you don’t.

OpenZFS 3.0 has three features that matter and one of them is that dedup is now a thing I would tell a colleague to use in production without adding a disclaimer.

RAIDZ expansion

This is the one that’s been on the wish list since people started migrating from hardware RAID controllers to ZFS in the mid-2000s. Linux’s mdraid could add a drive to a RAID5 array and redistribute data in the background. LVM had dynamic volume management. ZFS had: create a new pool, zfs send, swap the disks, zfs receive on the other end, hope nothing goes wrong for the sixteen hours the migration takes.

zpool attach now works on RAIDZ vdevs. You add a drive, the pool starts an expansion scrub, it redistributes data across the wider stripe in the background, and when it finishes your vdev has one more drive and more usable capacity. No migration. No downtime. No zfs send pipe that you’re nursing across a weekend.

The implementation took this long because RAIDZ is not software RAID-5. ZFS uses variable-width stripes — a write goes on however many disks it needs to, plus parity, and each block records its own stripe geometry. Expanding a RAIDZ vdev means re-striping existing data at the block level, updating per-block metadata, and doing all of this without dropping redundancy or corrupting the pool if you lose a drive mid-expansion. The engineering surface is larger than it looks. The FreeBSD-derived RAIDZ expansion code has been in review for the better part of three years. It is now in the release build and the scrub-based redistribution logic has been validated against the full pool testing suite.

Caveats: expansion is one drive at a time. You cannot go from RAIDZ-1 to RAIDZ-2 (different parity count — that’s a different problem). And the redistribution scrub puts sustained read-write load on the pool for the duration, which on a spinning-rust array full of old backups means you plan this for a maintenance window and not a Tuesday afternoon. But it works, it’s tested, and it’s the correct answer to “can I add a drive to my existing RAIDZ vdev.”

Fast dedup

The rewrite ships as a new on-disk DDT format (DDT v2) that you opt into explicitly — existing pools keep the old format until you migrate. The core change is that the DDT no longer needs to be fully resident in the ARC to function at acceptable performance. The new engine uses a two-level structure: a small hot index in memory for recently-seen checksums, and a cold table on disk for the long tail. Lookups against cold entries do a single disk read. Inserts that hit the cold table do a read-modify-write. The hot index size is configurable with a tunable.

What this means in practice: dedup with fast-dedup enabled on a pool with 40% dedup ratio now costs roughly the same ARC overhead as a pool without dedup. The IOPS cost of lookup isn’t zero — you’re paying disk reads on cold misses — but it’s bounded and predictable rather than the unbounded “the DDT grew to 48GB of RAM and now ARC has nothing left” failure mode of the old engine.

The recommendation changes from “don’t use ZFS dedup” to “use ZFS dedup if your data is actually dedup-friendly, enable fast-dedup DDT v2 format, set the hot index to something sane for your RAM budget, and benchmark it.” That’s a normal operational recommendation, not a warning label.

Backup pools with many copies of similar system images, VM datastores with cloned guests, or any pool storing large numbers of nearly-identical archives: fast dedup is the right answer and you can now say that without it being a career-limiting advice.

Block cloning

cp file1 file2 on ext4: read the data, write the data. On a 10GB VM image, that’s 10GB of reads and 10GB of writes.

Block cloning makes cp — or any copy operation at the syscall level via copy_file_range(2) — into a metadata operation for the data that’s already on disk. The filesystem records that two files share the same underlying blocks and marks them copy-on-write. No data movement happens until one of the files is modified. A cp of a 10GB image becomes a handful of metadata writes and completes in milliseconds.

This is what cp --reflink=always does on btrfs and XFS. ZFS users have been watching those filesystem announcements for years while ZFS had no equivalent. Block cloning in 3.0 is the equivalent, and it works across snapshots and clones, not just plain file copies.

The immediate beneficiary is any workflow involving large file copies: VM provisioning from a golden image, backup software that copies before modifying, deployment tooling that stages files before moving them into place. These operations are now cheap. The copy_file_range(2) syscall is what you want; tools that use read(2) + write(2) loops will not get the reflink behavior. Most modern copy utilities already use copy_file_range(2) on Linux when the destination filesystem reports it as supported.

What to do with this

OpenZFS 3.0 requires a kernel that supports the new DDT format if you’re migrating existing pools to fast dedup — read the release notes before upgrading pool feature flags, because feature flags are permanent and one-way. The usual ZFS upgrade advice applies: upgrade the software, verify the pool is healthy, then decide whether to enable new features.

FreeBSD packages are tracking the release. Linux distros are moving at the standard pace — Arch has it, the LTS distributions will take until their next point release cycle. If you’re on an LTS kernel with an old OpenZFS dkms package, you’re not getting 3.0 features until you update the out-of-tree module.

Block cloning and RAIDZ expansion are both default-enabled features on new pools. Fast dedup is opt-in because it requires the DDT v2 format upgrade.

Seventeen years to get usable dedup. Three years of RAIDZ expansion review. The ZFS development process is slow in the way that storage code should be slow — carefully, with extensive validation before anyone’s data depends on it. 3.0 is the version worth upgrading to.

See also: ZFS boot environments with bectl for the workflow I use before any pool upgrade, and ZFS send/recv replication for the migration pattern you hopefully won’t need to run after this.