The daemon wrote the files. The backup was empty.

The backup job had been reporting success for six months.

Backup success is a meaningless claim on its own. The only thing that matters is whether you can restore from it. Six months in, someone finally tried.

There was nothing to restore. The archive existed. It had a timestamp and a checksum and a reasonable file size. Inside: a valid tar archive, correct header, correct EOF marker, zero data files. Six months of daily backups, all successful, all empty.

The data was not gone. The application was running fine. Users were not complaining. The data files were on disk, being written to, correct in every way the application could observe. The application was writing. The backup was reading. The backup was reading nothing.

The directory looked fine

First thing: look at what the backup process was actually doing.

ls -la /var/data/appdata/
total 8
drwxr-xr-x  2 appsvc  appsvc  4096 Oct 10 03:12 .
drwxr-xr-x 18 root    root    4096 Apr 15 11:44 ..

Empty. The directory was empty.

But the application was writing to /var/data/appdata/. That was in the config. I checked the config.

find /var/data -name '*.dat' | head
/var/data/appdata/2026-10-10T03:12:45.dat
/var/data/appdata/2026-10-10T02:58:01.dat
/var/data/appdata/2026-10-09T23:47:22.dat

The files existed. ls -la on the directory showed nothing.

Wait.

ls -la /var/data/appdata/ 
# (empty)

sudo ls -la /var/data/appdata/
total 47812
drwxr-xr-x  2 appsvc  appsvc     4096 Oct 10 03:12 .
drwxr-xr-x 18 root    root       4096 Apr 15 11:44 ..
-rw-------  1 appsvc  appsvc   524288 Oct 10 03:12 2026-10-10T03:12:45.dat
-rw-------  1 appsvc  appsvc   524288 Oct 10 02:58 2026-10-10T02:58:01.dat
...

There were 4,217 data files in that directory. Without sudo they were invisible. The directory itself was readable and executable for everyone. The files inside were 0600.

The backup process ran as backupd. The data files were owned by appsvc, mode 0600. backupd could traverse the directory but could not read any of the files. tar dutifully ran, found 4,217 files it could not open, silently skipped them because nobody had passed --no-unreadable, and produced a valid archive containing nothing.

Six months of successful backups.

The permissions

0600 means the owning user can read and write. Nobody else. Not the owning group. Not world. Just appsvc.

The standard permissions for data files that a service writes are 0640 (owner read-write, group read, nobody else) or 0644 (owner read-write, everyone read). 0600 is fine for private configuration, credentials, keys. It is not fine for data files that a backup process needs to read.

The application itself had never requested 0600. It was calling open() with O_WRONLY|O_CREAT and mode 0666. The kernel applies the process’s umask to whatever mode you request at creation time. The process had been creating files with an effective mode of 0666 & ~umask.

With umask 022, you get 0644. With umask 077, you get 0600.

cat /proc/$(pgrep appsvc)/status | grep Umask
Umask: 0077

The application process had umask 0077. Everything it created was visible only to itself.

How it got there

The daemon had been running for 182 days.

ps -o pid,user,etimes,start,cmd -p $(pgrep appsvc)
  PID USER         ETIMES                 STARTED CMD
14832 appsvc     15724800 Apr 11 2026 /opt/appdata/bin/appsvc --config /etc/appdata/config.yaml

182 days. April 11. That was during the migration incident. I pulled up the incident log.

April 11: the old application server had been decommissioned early by mistake. The team scrambled to get the application running on the new host. Someone SSHed in, copied the binary, wrote a quick startup command, backgrounded it with nohup, logged out. The application had been running on that nohup ever since. No init script. No supervisor. No service file. nohup ./appsvc &, disown, done.

The person who started it had umask 077 in their .bashrc. Good practice for a personal account. Bad default to silently hand to a production daemon.

nohup preserves the shell environment, including umask. The daemon inherited 0077 at launch and had been creating 0600 files ever since. Nobody checked. Nobody noticed. The first write was at 00:04 on April 12. The first backup ran at 02:00 on April 12. Eighteen hours into the daemon’s life, it had already produced data the backup could not read.

What the daemon should have done

A well-behaved Unix daemon resets its own umask early in startup. This is in the original Unix daemon recipe, the one from Stevens, the one that every sysadmin who grew up on real Unix internalized. The checklist:

1. Fork and have the parent exit (detach from the terminal session)
2. Call setsid() to become a new session leader
3. Change directory to / so you do not hold any mount points open
4. Reset umask to a known value, typically 0022 or 0027
5. Redirect stdin/stdout/stderr to /dev/null

Step four. The application code had steps one through three. It never set umask. It trusted the environment it inherited, which is exactly the wrong thing to trust when you’re a long-running production process.

The umask(2) call is one line:

umask(022);

Or, if the application is written in Go:

syscall.Umask(022)

Or Python:

os.umask(0o022)

Or you set it in the startup wrapper before exec-ing the binary. If you’re using an init system or supervisor that does not do this for you automatically, put it in the run script:

#!/bin/sh
umask 022
exec /opt/appdata/bin/appsvc --config /etc/appdata/config.yaml

One line. The daemon would have had 0644 files from day one and this conversation never happens.

The backup gap

The good news: the data itself was intact. The application had been writing correctly to its own files for 182 days. The problem was access, not corruption.

The immediate fix was changing permissions on the existing files:

find /var/data/appdata -name '*.dat' -exec chmod 640 {} +

4,217 files. Added group read. The backup process ran as backupd, which was in the appsvc group. After the chmod, backupd could read the files.

Before doing a restore test, we needed an actual backup. Ran the backup job manually. This time: 4,217 files, 2.1 GB. Restored to a test directory. Data checked out.

The six months of empty archives cannot be recovered. They are archives of nothing. If the data from that period had been lost, we would have had no recovery path. The data was not lost because the application kept it, but we had been relying on a backup that could not have helped us if we had needed it.

The startup situation

The daemon running for 182 days under nohup with no supervisor was the other problem. A daemon running under nohup does not restart if it crashes. It does not restart if the machine reboots. It does not appear in any service inventory. systemctl status knows nothing about it. service knows nothing about it. The only way to find it is ps aux or noticing that something stopped working.

This machine had not been rebooted in 182 days. If it had, the application would have come up missing and nobody would have known until a user complained, which could have been hours.

We wrote a proper service unit:

[Unit]
Description=AppData Service
After=network.target

[Service]
Type=simple
User=appsvc
Group=appsvc
UMask=0027
ExecStart=/opt/appdata/bin/appsvc --config /etc/appdata/config.yaml
Restart=on-failure
RestartSec=10

[Install]
WantedBy=multi-user.target

Note UMask=0027. systemd units support this directive directly. Files the service creates will be 0640 for directories and 0750 for executables. If you’re using systemd, this is the right place to set it: in the unit file where anyone reading the unit knows what the service’s file creation mode will be.

Rod uses runit. For runit the setup is:

#!/bin/sh
# /etc/sv/appdata/run
exec chpst -u appsvc:appsvc umask 027 /opt/appdata/bin/appsvc --config /etc/appdata/config.yaml

chpst sets user, group, and environment before exec. Clean, explicit, auditable. The process that keeps accepting connections would also benefit from chpst -o 65536 to set the file descriptor limit before the service starts.

The restore test

The industry-standard advice is: test your restores regularly. This is genuinely good advice that almost nobody follows because testing restores is boring and requires finding time you do not have and nobody ever gets credit for it. You only get credit for running a backup that works. Whether it works is not checked until you need it, and by then you’re in an incident, and the discovery that the backup is empty is the worst possible time to make that discovery.

The way to make yourself test restores is to require a restore to close the backup ticket. Not “backup ran successfully” as the completion criterion. “Restore completed and data verified” as the completion criterion. The difference in policy is a few words. The difference in operational reality is the six months of empty archives we found instead of the six months of empty archives we did not find.

We now have a weekly restore test to a scratch directory with a checksum comparison against a known-good sample. It runs in CI. The backup job’s status badge is green only if the restore test passed.

The backup has been running correctly for three weeks. First time in eight months that the backups have been correct. The config file that changed while nothing reloaded was visible the moment something stopped working. Six months of incorrect backups were invisible until someone asked whether they could actually use one.

The umask in your dotfiles

umask 077 in a .bashrc is not wrong. For an interactive user session, files you create being readable only by you is reasonable default behavior. Private keys, configs, scratch files, whatever you produce in a shell session: 0600 is fine.

The problem is umask propagates across fork and exec. Any process you launch from that shell inherits your umask. If you start a service from your shell session, it gets your umask. If that service stays running after you log out, it keeps your umask. The service does not know it was started with an unusual umask. It just creates files the only way it knows how, with whatever mode is in effect, and the mode in effect is yours.

This is a systems programming assumption that bites people who are not thinking about it: the process environment is not neutral. It is whatever the launching environment was. You are responsible for explicitly resetting anything your process cares about: umask, ulimit, working directory, signal handlers, file descriptor flags. The listen queue was a number nobody had set intentionally. This was a value set intentionally for the wrong scope, propagated further than intended, doing quiet damage for six months.

Reset your umask in your daemon startup. It’s one line. Do it before you have six months of empty archives explaining why it matters.