The ticket said the app was down. The process list said it was up. Both were true, which is the kind of afternoon I used to bill extra for.
The app’s front end talked to a small backend daemon over a Unix domain socket. ps showed the daemon alive, eleven days of uptime, memory flat, CPU bored. The front end logged one line over and over:
connect() to unix:/tmp/appd.sock failed (2: No such file or directory)
ls /tmp/appd.sock agreed with the error. No such file. And the daemon had been running since before the file vanished, happily listening on something that no longer had a name.
What unlink does to a listening socket
A Unix socket has two lives. There is the kernel object, the thing the daemon bind()ed and listen()ed on, and there is the directory entry that lets other processes find it by path. unlink() removes the second and leaves the first alone. The daemon holds a file descriptor, the descriptor is still valid, accept() still blocks politely. Nothing in the kernel tells the daemon its name is gone.
Clients are different. connect() resolves the path, and a path that no longer resolves is ENOENT before the socket is ever consulted. Connections that were already established keep working, which is why the first symptom was intermittent: long-lived clients were fine, anything that reconnected failed. Same shape as the service that was healthy and accepting no new connections, except here the listener was fine and the door had been bricked over.
You can see the orphan from the daemon’s side. On FreeBSD, fstat -p <pid> lists the socket descriptor, and a socket bound to a path that has since been removed shows up without a usable name. On Linux, ss -xlp shows the listening socket and its original path, and ls -l /proc/<pid>/fd marks a deleted path for file-backed descriptors. The tooling disagrees on presentation, but the pattern is the same: the daemon thinks it is listening on a path that ls cannot find.
Who deleted it
Nobody on the team. I checked shell history, deploy logs, and the config management run. Clean.
The socket lived in /tmp because the original author had written /tmp/appd.sock as a default and nobody ever overrode it. And the host had a periodic cleaner that removes old entries from /tmp. These are common: FreeBSD’s periodic has daily_clean_tmps_* knobs, and many Linux distributions ship a tmpfiles cleaner with an age cutoff. The details differ by OS, version and local config, so read yours. Whether they touch sockets, and by which timestamp, is exactly what bites you.
The thing that makes this nasty is that “age” is judged by timestamps on the directory entry, and a socket that is only ever connect()ed to does not necessarily look recently used to a cleaner that checks atime or mtime on the inode. A busy socket can look as stale as an abandoned one. Eleven days in, past the cleaner’s cutoff, off it went.
I confirmed it the boring way: the cleaner’s config on this box had a retention in days, the socket’s creation time was just past it, and a second host with an identical daemon but a fresher restart still had its socket. That is not proof of the exact trigger, but it is the only theory that fit every observation, and the fix below removes it either way.
The fix
1. Restore service. Restarting the daemon makes it bind() again and recreate the file. Do that last, after you have captured the evidence, because you will want to show someone the orphaned descriptor. Do not touch or recreate the path by hand; a regular file at that name makes bind() fail with EADDRINUSE on the next start.
2. Put the socket where nothing sweeps. Runtime state goes in a runtime directory owned by the service: /var/run/appd/ on BSD, /run/appd/ on Linux, created by the rc script or unit with the right owner and mode. /tmp is for things you are happy to lose. A socket your front end depends on is not one of them. This is the same lesson as the tuning that lasted until the reboot: state that lives in the wrong place has a lifetime nobody wrote down.
3. Make the daemon notice. The cheap version is a health check that does what the client does: open the path with connect(), not pgrep. The check that said “process alive” had been green the entire outage. I wrote about health checks that test the wrong service a while back and apparently did not read myself.
4. Optional, if you control the code. Have the daemon stat() its own socket path periodically and exit loudly if the name is gone or no longer points at its inode. A supervisor restarts it, the name comes back, and the outage becomes a blip with a log line instead of an afternoon.
The rule
A running process and a reachable service are different facts. The kernel keeps the first alive long after the filesystem has ended the second. If clients find you by path, the path is part of your uptime, and /tmp is a bad place to keep any part of your uptime.
More like this live in the reliability hub.
