Every ZFS replication failure I have been paged for has the same cause: the incremental needs a common snapshot on both sides, and somebody’s retention script deleted it. Not a bug in ZFS. A prune job, working as designed, removing the one snapshot the next send was standing on.
Two small features fix this properly: bookmarks and holds. Neither is exotic. Both are in the man page for zfs(8) and almost nobody uses them.
The problem in one paragraph
An incremental zfs send -i pool/data@a pool/data@b needs @a on the sender, and the receiver needs @a too. If you snapshot hourly and prune to the last 24 on the source, but the replica has been unreachable for two days, @a is gone from the source. Your only option is a full send. On a few terabytes over a slow link, that is a bad weekend. I wrote up the basic flow in ZFS send/recv replication; this is the failure mode that post does not protect you from.
Bookmarks: the snapshot’s name without its data
zfs snapshot pool/data@2026-11-07
zfs bookmark pool/data@2026-11-07 pool/data#2026-11-07
zfs list -t bookmark
A bookmark is a pointer to the transaction group where a snapshot was taken. It holds no data blocks, so it costs next to nothing and does not pin any space. Bookmarks use # where snapshots use @. They need the bookmarks pool feature, which is enabled by default on any pool created in the last decade.
The trick is that an incremental send can start from a bookmark:
zfs send -i pool/data#2026-11-07 pool/data@2026-11-09 | ssh replica zfs recv tank/data
The sender no longer needs the old snapshot. It can destroy @2026-11-07 to reclaim the space, keep the bookmark, and the next incremental still works, provided the receiver still has @2026-11-07. Bookmark after each successful replication, delete the old snapshot, keep one bookmark as the anchor. That is the whole pattern.
Holds: make destroy refuse
A bookmark protects the sender’s ability to send. It does nothing about the receiver, or about a snapshot you are mid-send from. For that, holds:
zfs hold keep pool/data@2026-11-07
zfs holds pool/data@2026-11-07
zfs destroy pool/data@2026-11-07
The destroy fails with a “dataset is busy” error while any hold exists. The tag (keep here) is arbitrary; use something that says who placed it, such as replica-b, so the next admin knows which job to blame. Release with zfs release keep pool/data@2026-11-07. Clones also keep a snapshot from being destroyed, but a hold is the explicit version and shows up in zfs holds.
zfs destroy -d marks a snapshot for deferred destruction: it goes away when the last hold or clone disappears. Useful when a prune script should state its intent without racing a running send.
Put the hold on the receiver’s newest snapshot too if anything else prunes there. Retention scripts do not know about each other, and they never will.
What a bookmark cannot do
- You cannot roll back to a bookmark, mount it, or read files from it. It is a send source and nothing else.
- It does not help if the receiver also lost the base snapshot. The incremental stream is relative to
@2026-11-07, and the receiver must have it, unmodified. If you wrote to the replica’s dataset since, you needzfs recv -For a rollback first. - It does not follow a rename of the dataset the way you might hope. Check
zfs list -t bookmarkafter any restructuring rather than assuming. - Bookmarks are created from snapshots of the same dataset, and they go away with the dataset.
- Holds are per snapshot. Prune scripts that destroy recursively will hit the busy error and, depending on how they were written, either log it and move on or abort halfway. Read yours before you rely on this.
The retention script I actually run
After a successful recv, in this order: bookmark the snapshot just sent, release the hold on the previous one, destroy the previous snapshot, destroy the previous bookmark. If any step fails, stop, so you never end up with no anchor. This is the same “never delete the last known-good thing first” rule behind bectl boot environments, and the same reasoning that makes the backup that restored perfectly a story worth reading before your next prune job.
More on the filesystem side lives in the ZFS and storage hub. Test the whole loop on a throwaway pool with a file-backed vdev before you point it at anything with a name.
