$RodHat_
Console Tips

zfs send is the backup tool you already have and probably aren't using

Published by

zfs send is the backup tool you already have and probably aren't using
Photo: AI-generated — no human photographer / RodHat AI Cover

Every file-level backup tool starts by walking the tree and stat’ing everything to find out what changed. On ten million files that walk costs more than the transfer. ZFS already knows what changed — it’s a copy-on-write filesystem, the delta between two snapshots is a set of block pointers it can enumerate directly — so zfs send -i reads the changed blocks and nothing else.

No walk. No timestamp comparison. No “did the mtime lie to me.” The filesystem hands you the diff.

The whole thing

# on the source
zfs snapshot -r tank/data@2026-07-31
zfs send -R -I tank/data@2026-07-30 tank/data@2026-07-31 \
  | ssh backup-host 'zfs recv -Fdu backup/tank'

That’s a complete incremental replication of a dataset and all its children.

Flags that earn their place:

  • -R replicate: children, snapshots, and dataset properties come along. Without it you’re sending one dataset and losing the properties, which means the receiving side has different compression and recordsize than the source and you find out during the restore.
  • -I (capital i) sends all intermediate snapshots between the two named. Lowercase -i sends only the delta between exactly those two, discarding the ones in between. If you take hourly snapshots and replicate daily, -I preserves the hourlies on the target; -i doesn’t. Pick deliberately.
  • -F on receive: force a rollback of the target to match. The target must be a superset-free descendant of the source snapshot; if anything wrote to the target since — even an atime update from someone running ls — the receive fails without it.
  • -u don’t mount the received filesystem. On a backup host you want the data, not forty datasets mounted over your directory tree. This one saves real embarrassment.
  • -d strip the source pool name so tank/data lands at backup/tank/data.

Set readonly=on on the receiving datasets. It’s the cheapest possible protection against the -F rollback problem, and it costs nothing.

The resume token

This is the feature that changes how you feel about large transfers.

A 3TB send dies at 2.7TB because the WAN blipped. Historically: start over. Now:

# on the receiver, after a failed transfer:
zfs get -H -o value receive_resume_token backup/tank/data
1-e1f2a3b4c-...
# on the sender:
zfs send -t 1-e1f2a3b4c-... | ssh backup-host 'zfs recv -s backup/tank/data'

It picks up where it stopped. The -s on the receive side is what makes the token get saved in the first place — put it on every receive you ever write, permanently, because you cannot retroactively want it after the transfer dies.

Abort a stale one with zfs recv -A backup/tank/data, which you’ll need when you want to send a different stream and the dataset is still holding a token.

Verification, and what “verified” means

The stream carries checksums, and zfs recv validates them. If the stream is corrupted in transit, the receive fails — it does not silently write bad data. That’s a genuinely different guarantee from rsync, which will happily reproduce whatever it read.

Combine with a scrub on the target and you have an actual answer to “is the backup good,” which is more than most backup systems can honestly claim. Then go one further and test the restore, because a verified backup of the wrong dataset is still the wrong dataset, and I have watched that discovery happen live.

Raw sends, for encrypted datasets

zfs send -w -I tank/enc@old tank/enc@new | ssh backup 'zfs recv -su backup/enc'

-w (--raw) sends the blocks still encrypted. The receiving side stores ciphertext and never has the key — it cannot read your data, and it doesn’t need to in order to store it, scrub it, or send it back. That’s the correct model for offsite backup to hardware you don’t control, and it’s a single flag.

Caveat: the target can’t do anything requiring plaintext, obviously. And you must have the key to restore, so back up the key somewhere that is not the encrypted pool. People forget that one exactly once.

The pipeline in practice

zfs send -Rw -I "@$LAST" "tank/data@$NOW" \
  | zstd -3 -T0 \
  | ssh -c aes128-gcm@openssh.com backup 'zstd -d | zfs recv -Fsdu backup'

zstd in the middle because zfs send output compresses well even when the dataset is compressed on disk (the stream includes metadata and holes). aes128-gcm because it uses AES-NI and on a 10Gb link the default cipher can genuinely be your bottleneck — and connection reuse keeps the per-run setup cost off the clock.

Add mbuffer between stages if the link is bursty; it smooths the stall-and-surge pattern that makes zfs recv sit idle waiting on the socket.

What it isn’t

zfs send replicates a dataset. It is not a file-level restore tool — you cannot pull one file out of a stream without receiving the whole thing. That’s fine, because on the receiving side you have real snapshots and .zfs/snapshot/ gives you every file directly. But it means the stream itself is not an archive format, and storing raw .zfs-send files on object storage instead of receiving them into a pool gives up the verification, the resumability, and the browsability all at once. People do it anyway. Don’t.