Every few weeks someone asks why I’m so rude about “it’s just a mv.” Here is a Tuesday that explains it.
A load balancer started handing out empty upstream lists. Not for long, a few hundred milliseconds at a time, a couple of times an hour. Enough to throw a burst of 502s into the graphs and make the on-call question their life choices. Nothing had deployed. Nothing had restarted. The config was correct every time anyone looked at it.
The publisher
A small script generated the upstream list from service discovery and published it where the balancer’s reload hook watched. Roughly:
gen_upstreams > /tmp/upstreams.conf.new
mv /tmp/upstreams.conf.new /etc/lb/upstreams.conf
Write to a temp name, mv over the real one. This is the correct pattern, and I’d have defended it in court. rename(2) within one filesystem is atomic: a reader sees the old file or the new file, never a mix. The script had run that way for two years without incident.
The balancer used inotify on the file and re-read it on any change. Fast, no polling, no race. Except there was a race, and a reasonably bright intern could have found it, if the intern had been given the right mount table.
What changed
Three weeks earlier, a hardening pass had put /tmp on tmpfs. Sensible. Noexec, nosuid, size-capped, gone at reboot. Nobody connected it to the publisher, because the publisher never mentioned a mount.
Now the temp file lived on tmpfs and the destination lived on the root filesystem. Different devices. rename(2) on different filesystems fails with EXDEV, “invalid cross-device link,” and it fails cleanly, which is what makes it a good primitive.
But mv is not rename(2). mv is a program, and when it gets EXDEV it shrugs and does the thing you didn’t ask for: open the destination with truncation, copy the bytes across, unlink the source. Same exit code zero. Same command. Different semantics. You can see it for yourself:
strace -f -e trace=rename,renameat,renameat2,openat,unlink mv /tmp/a /var/b 2>&1 | tail
You get a renameat2 returning EXDEV, then an openat on the destination with O_TRUNC, then writes, then an unlink of the source. The window between the truncate and the last write is the window where the file is empty or short.
inotify fired on the truncate. The balancer woke up, read a zero-byte file, and dutifully configured zero upstreams. The second inotify event, the write completing, fixed it a few hundred milliseconds later. That’s the whole outage. The size of the window was a function of file size and disk mood, which is why it only showed up a couple of times an hour, when the generated list was large enough to span more than one write.
Why nobody saw it
Because the script worked. mv returned zero. The file at rest was always right. Every check anyone ran on the result passed, since every check ran after the copy had finished. The bug lived in a window nothing was looking at, and the only observer was a daemon whose job is to act on whatever it reads.
The mount change and the publisher were reviewed by different people, in different quarters, neither of whom could see the other’s half. I’ve written about this shape before, in the config file that changed and nothing reloaded. Systems rarely break at the thing you edited. They break where an assumption you never wrote down met a change somebody else made.
The fix
Put the temp file on the same filesystem as the target, ideally the same directory, so the rename can’t cross a device:
tmp=$(mktemp /etc/lb/.upstreams.conf.XXXXXX) || exit 1
gen_upstreams > "$tmp" || { rm -f "$tmp"; exit 1; }
sync "$tmp" 2>/dev/null || true
mv -T "$tmp" /etc/lb/upstreams.conf
A few notes, because each line is there for a reason:
mktempin the destination directory. Same directory means same filesystem. Always. (Bind mounts and overlay oddities can still bite, but this removes the common case.)- GNU
mv -Ttreats the target as a plain file, and GNUmvhas no flag to forbid the copy fallback. If you need a hard guarantee, callrename(2)from something that exposes it directly (a ten-line C or Perlrename()call) and let it fail loudly withEXDEVinstead of improvising. fsyncthe file before the rename if you care about the result surviving a power cut. On ext4 and XFS the replace-via-rename pattern is mostly protected by heuristics, but “mostly” is a word I distrust. Flush the data, rename, then fsync the directory if the publisher has to be durable.- Have the generator refuse to publish a result that is empty or implausibly short, and have the reader refuse to apply one. A balancer that accepts a zero-length upstream list as valid configuration is a balancer that will eventually be told to do exactly that.
If the reader is yours, make it ignore events for the temp name and react only to the rename into place, so a half-finished copy can’t wake it at all. See also the set-operations tip on comm and join for the sort of small tool contract that bites the same way, and the Unix shell and tooling hub for the rest of the scripts that lie politely.
What I’d tell the hardening team
Nothing about putting /tmp on tmpfs was wrong. Do it. But it is a change to every program that has ever used /tmp as a staging area, and “which of our scripts stage there and then move out?” is a grep you can do in ten minutes. grep -rn '/tmp' /usr/local/bin /etc/cron* would have found this one on day one.
mv is atomic when it’s a rename and a copy when it isn’t, and it will not tell you which. Write your own scripts as if you were the one who has to explain the 502s.
