The temporary NFS mount that ran production for six years

There’s a specific flavour of dread that comes from running mount on a production host and seeing something you don’t recognise.

Not something misconfigured. Something you have never heard of, that has clearly been there a long time, that nothing in any document mentions, and that half your application depends on.

How it started

I wasn’t there for the beginning, but the archaeology was straightforward once we knew to look.

Six years earlier, a product launch was two days out and a feature needed shared file storage between four application servers. The proper answer — object storage, an API, a small amount of application work — was maybe a week. They had two days.

So somebody took a spare box, installed the NFS server package, exported a directory, mounted it on the four app servers, and put a line in /etc/fstab. Twenty minutes of work. Launch happened. Everyone moved on.

I want to defend that person. Given two days and a launch, that is the correct call. Shipping beats architecture. The failure wasn’t the decision — it was that nothing ever came back to it, and there was no mechanism by which anything ever would.

The six years

The tickets existed. I found them: three of them, filed at various points, all titled some version of “migrate shared storage off the temporary NFS box.” Each one was opened by somebody who’d noticed, discussed briefly, deprioritised, and eventually closed as stale by a bulk backlog cleanup.

In the meantime:

More things started using it. Because it was there, and it was easy, and mounting an existing share is a one-line change while provisioning proper storage is a conversation with a platform team. By the time we found it, eleven services were reading from it and four were writing.

It survived a datacentre move. Because the migration was done by replicating hosts, and the NFS box was a host, so it got replicated. Nobody asked what it did. It came up on the other side and everything kept working, which is exactly the outcome that guaranteed nobody would look.

It survived an acquisition. Two infrastructure teams merged, one inventory was reconciled against the other, and this box appeared in neither — it had been provisioned outside the normal process and had never been added to the asset register. It was in the racks and on the network and in nobody’s spreadsheet.

Everyone who knew left. Not dramatically. Normal attrition over six years. The last person who could have told you what it was for moved to a different company in year four.

Finding it

We found it because it ran out of disk.

The application started throwing write errors, which surfaced as a feature quietly failing, which surfaced as a support ticket, which eventually got to somebody who read the actual error message instead of the summary. The error said No space left on device and named a path nobody recognised.

So we went to the box the path pointed at. And then we spent about four hours doing something I have never had to do before or since: figuring out, from first principles, what a production server was.

No documentation. No ticket. Not in the CMDB. Not in configuration management — it had been set up by hand and had never been brought under it. No monitoring beyond a basic ping check inherited from a subnet-wide rule. No backup, because backups were configured per-service and this box belonged to no service.

The hostname was fileserv-tmp-01. The tmp was doing an enormous amount of work in that name and had been for six years.

What it was holding

We worked it out by watching. fuser on the mount points across the app servers, then reading code paths for each service that appeared.

Generated PDF documents users could re-download for up to seven years, for compliance reasons. Uploaded images referenced by database rows that stored a path but not the content. A shared cache that four services treated as authoritative because it had never been empty. And — my favourite — a directory of CSV exports that a nightly job read, produced by a job on a different host that wrote into the same share, which meant two services were coordinating through a filesystem with no locking whatsoever.

Roughly 900GB of data, none of it backed up, all of it required, sitting on a single machine with a single disk pair and no redundancy of any kind.

The disk had filled because the log directory of the NFS server itself was on the same volume, and it had been growing for six years with no rotation.

The bit that scared me

We ran smartctl on the drives.

One of them had been reporting reallocated sectors for — we could only estimate from the power-on hours and the counter — probably two years. There was no monitoring on it, so nothing had ever said anything. Two years of a degrading drive in a mirror, on a box holding 900GB of unbacked-up production data that eleven services depended on.

If that mirror had lost its second drive we would have lost seven years of compliance documents and had no path to recovery. None. Not a slow recovery — no recovery at all. The data existed in exactly one place.

That’s the number I still think about. Not the six years. The fact that we were, for some unknown length of time, one hardware failure away from an unrecoverable loss, and no system anywhere in the company was capable of telling us.

Getting out

Twelve weeks, in the end, which felt slow and was probably right.

Backups first, that same day — an rsync to somewhere with actual retention, before anything else, because everything else was optional and that wasn’t. Then monitoring on the box, then a disk replacement, then the actual migration service by service to object storage, then decommissioning.

The migration was mostly unglamorous and one part was genuinely hard: the two services coordinating through the shared directory had an implicit protocol based on filesystem semantics — one wrote a file, the other polled for it and deleted it when done. Reimplementing that on object storage without a queue meant reasoning carefully about what happens when both run at once, which nobody had ever done, because NFS’s behaviour had accidentally been good enough.

What I actually changed afterwards

Not “document everything.” That’s the answer everyone gives and it doesn’t work — documentation decays, and it decays fastest for the things nobody owns, which are precisely the things at risk.

Two things that did work:

An unowned-asset sweep. Quarterly, automated: every host that responds on the network, cross-referenced against the asset register and configuration management. Anything present in reality and absent from both gets a ticket with a named owner and a two-week deadline. It found three more things in the first run. One of them was another tmp box.

A rule about the word temporary. Anything named or described as temporary gets an expiry date at creation and a calendar entry on that date. Not a promise to revisit — a dated event with a person’s name on it. When the date arrives, either it’s been replaced or somebody explicitly renews it and takes ownership.

That second one is soft and I was sceptical of it. It has worked better than I expected, for a reason I didn’t anticipate: it’s not the reminder that does the work. It’s that assigning a name and a date at creation time makes people notice, right then, that they’re creating something permanent. About half the time they just do it properly instead.

The general shape

Every organisation has one of these. Probably several. The characteristics are consistent:

It was created under time pressure by a competent person making a reasonable call. It was never brought into the normal processes, because it was going to be replaced. It accumulated dependents because it was convenient. It outlived everyone who knew about it. And it is invisible to your monitoring, your inventory, and your architecture diagrams — all of which describe the system somebody designed, rather than the system that actually exists.

Go run mount on a production host. Then crontab -l as every user on it. I’d genuinely like to be wrong about this, and I haven’t been yet.