The setup was a classic. Live config at /etc/app/app.conf, which was a symlink into a release directory: /srv/app/current/conf/app.conf. Deploys swapped current to a new release and the symlink followed along. Tidy, boring, correct.
Someone needed to flip a timeout during an incident. They ran sed -i 's/timeout = 5/timeout = 30/' /etc/app/app.conf, checked the file with cat, saw 30, restarted the service, and went to get coffee. The timeout stayed at 5 in practice for a week. Then a deploy went out, and the 30 vanished entirely, which is what finally got somebody’s attention.
What sed -i actually does
sed -i does not edit a file. It writes a new temporary file in the same directory, then renames it over the original name. That is why it’s safe against a crash halfway through and also why it has side effects people forget.
The path you hand it is /etc/app/app.conf. The thing at that path was a symlink. The rename replaced the symlink itself with a brand-new regular file in /etc/app/. The release directory’s copy, the one the symlink pointed at, was never touched.
So there were now two configs. The one the service read, which had 30. The one in the release tree, which still said 5. The next deploy re-created the symlink over the regular file, and the service went back to the release copy. A change that only existed on the live path died on the next deploy, exactly as a change to a non-source-of-truth should.
If this smells like the rename that was atomic except the one time it wasn’t, it’s the same machinery from the other side: rename replaces a name, not the thing the name used to point at.
GNU sed has --follow-symlinks for this. Read man sed on your own box, because the behavior and flag differ across implementations. FreeBSD’s sed also wants an explicit backup suffix argument with -i (empty string for none), so a command pasted from a Linux answer either errors or eats the next argument as a suffix. Check, don’t assume.
The second half nobody mentions: hard links
The same mechanism breaks hard links. If two names share an inode and you sed -i one of them, that name now points at a new inode and the other name keeps the old contents. People who use hard links for config sharing or for rsync --link-dest snapshot trees get this one in the face, usually quietly.
Ownership and mode can also drift. The new file is created by the user running sed, with permissions derived from the original where the implementation bothers to copy them. Run it as root on a file owned by an app user and verify the result with ls -l before you decide it worked.
How I’d have caught it in thirty seconds
Before editing anything on a path you didn’t create, look at what it is:
ls -l /etc/app/app.conf
readlink -f /etc/app/app.conf
If readlink -f prints something different from the path you typed, edit the real target, or use the tool’s symlink-following option on purpose. After editing, run ls -l again. A file that used to start with l and now starts with - is the whole bug.
To hunt for damage across a tree, compare what is a regular file against what your deploy tooling thinks should be a link. Anything under a managed directory that deploy expects to own and is now a plain file is a suspect.
The fix and the rule
The immediate fix was editing the file in the release source, committing it, and deploying. Slower than sed -i and the only way the change survives. The real lesson was the same one as the config file that changed and nothing reloaded: “the file looks right” and “the system is using the right file” are two separate claims, and you only verified one.
My rule: never hand an in-place editor a path I haven’t resolved. Quick edits during an incident are fine, but a hot edit that isn’t also made at the source is a loan, and the deploy is the collection agent. More tales in this vein, where the thing looked healthy and wasn’t, are in the reliability hub.
