$RodHat_/search
Rod's Tales

The restore worked and every file belonged to a stranger.

Published by

This article may contain Amazon affiliate links. I may earn a commission if you buy through them. Disclosure.

Editorial card: The restore worked and every file belonged to a stranger.
Editorial card: RodHat
On this page · 5 sections
  1. What I did
  2. What tar actually stores
  3. Names versus numbers
  4. The fix
  5. Ownership is part of the data

The disk died on a Tuesday. The restore took forty minutes, tar exited zero, checksums matched, and the application came up. Then the mail daemon refused to read its own spool, the database complained about its data directory, and a cron job that had run for six years started mailing Permission denied to nobody.

Every file was there. Every byte was right. Every owner was wrong.


What I did

The replacement box was a fresh install, because that’s what you do with a dead disk. I created the service accounts by hand from a checklist, unpacked the backup as root, and went to get coffee.

cd / && tar -xpf /mnt/backup/home-and-var.tar

That -p is the part I was proud of. Preserve permissions. I had been burned by that before. What I had not thought about was what tar does with ownership when it’s running as root.


What tar actually stores

A tar archive records both a numeric uid/gid and the user and group names for each member. On extraction as root, GNU tar defaults to --same-owner and, when the archive has names, it looks the names up on the local system and uses the local numbers. Only if the name doesn’t exist does it fall back to the stored number.

That is a feature. It’s the right default when you’re moving a tree between two machines with different uid assignments. It is exactly the wrong thing to be surprised by at 02:00.

On the old host, postgres was uid 70 and mail was 8. On the new one, I had created the accounts by hand in a different order, so postgres was 1002 and a service called queue had taken 70. tar looked up postgres, found 1002, and chowned everything to 1002. Which was correct by name. But the database’s config, the init scripts, and a bind mount from a second disk that survived the crash all still said 70, because those were stored as numbers on a filesystem that never went through tar.

So the restored tree was consistent with the names and inconsistent with everything that wasn’t restored. The survivor disk’s files now belonged to queue. A queue daemon had write access to a live database directory for about ten minutes before I noticed. It didn’t write anything. I got lucky and I know it.


Names versus numbers

The kernel doesn’t know names. It knows numbers. /etc/passwd is a lookup table that userland consults, and a restore is the moment two lookup tables meet.

Two defensible policies exist:

  • Restore by name (the GNU tar default as root). Right when you’re migrating to a new layout and account names are the stable identity.
  • Restore by number (--numeric-owner). Right when you’re rebuilding the same host, or when anything outside the archive references uids directly: other disks, NFS exports, container volumes, ACLs.

I wanted the second and got the first. rsync has the same split in --numeric-ids, and the same default of translating by name. If you’ve read the rsync flags that matter, this is the one I’d have underlined harder.


The fix

For the restore:

tar --numeric-owner --same-owner -xpf backup.tar -C /

Before the restore, the part I skipped: dump the uid/gid map from the old host into the backup itself, next to the data, and compare it with the new box before extracting anything.

getent passwd | awk -F: '$3 >= 100 {print $1, $3, $4}' | sort > ids.txt

On the new host, diff it against the same command. A mismatch in that file is a five-second find. The same mismatch found after extraction is an afternoon of find / -uid 1002 -exec chown, with the risk of chowning something that was legitimately 1002.

And pin the service uids in your provisioning, don’t let install order decide. Hand-created accounts are how postgres became 1002.


Ownership is part of the data

I’ve said before that a backup you haven’t restored is a rumour. This is the rider: a restore you haven’t inspected is a rumour with a checksum. sha256sum told me the contents were identical, and it told me nothing about who was allowed to read them. Metadata isn’t covered by hashing file bodies, and the metadata is where the permissions live.

I ran a find for anything owned by a uid with no passwd entry, and another for anything whose owner didn’t match the directory above it, before declaring the box done. Both take seconds. I’d also read the file root couldn’t touch a month earlier and still did not think to look at who owned the thing.

Nothing was lost. Nothing was corrupt. Everything was just owned by someone else. That’s a worse class of bug than a missing file, because it passes every test that only checks whether the file exists.