The number 1024 was not a coincidence

Two in the morning. My pager goes off. The service is dropping connections — intermittently, only at peak load — and the monitoring dashboard says everything is fine. CPU: fine. Memory: fine. Network throughput: fine. Error rate: fine.

Clients are getting Connection refused. The monitoring system samples at sixty-second intervals, which is long enough for a burst of failures to average out and disappear. We were flying in the window between samples, looking fine, dropping real traffic.

I do the thing you do at 2am: I find a machine in the affected cluster and start looking at the process with my own hands.

lsof -p $(pgrep servicename) | wc -l

ulimit -Hn on the running process: 1024.

There it is. The process was three file descriptors away from its hard ceiling, and when a new connection came in, the kernel said no, and the client got EMFILE translated to a connection refusal, and the monitoring sampled between bursts and saw a clean run, and the alert never fired.

Fix: restart the process with ulimit -n 65536. Done. Thirty seconds, total. Back to bed.

I did not go back to bed.

1024

The next morning I started pulling on the thread. Where was the 1024 coming from? Not from the service’s own config — the service binary doesn’t set its own FD limit. Not from the process supervisor config — I’d checked that, it was clean. Not from /etc/security/limits.conf — that file was fine, set to 65536 for the service user.

I found it in a startup wrapper script. A shell script that predated the process supervisor by years, that the supervisor was calling as an entrypoint, and that had at the top:

#!/bin/sh
ulimit -n 1024
exec /usr/local/bin/servicename "$@"

The comment above the ulimit line read: # Limit open files — sockets need FDs.

No number. No rationale for the specific value. Just 1024, sitting there like it was handed down from Sinai.

I looked at the git history for that script. The script had been imported from the previous infrastructure as-is during the migration. The commit message: port init script from old cluster. Before that I had to go to a different repository — the old cluster’s config repo. Same script, same ulimit -n 1024, different git history. That commit message: migrate service configs from bare metal era.

Bare metal era. There’s a phrase. I found the original bare-metal script in an archive tarball that someone had thoughtfully committed to the config repo under legacy/. Same line. Commit from 2008: initial sysv init script.

I went further back. Under legacy/archive/, there was an older tarball. 2003. Same script. Same ulimit -n 1024. The comment on that version read: # 1024 should be enough, most machines cap here anyway.

Machines in 2003 often had a kernel default of 1024 for the per-process FD limit. The person who wrote this in 2003 was not being reckless — they were noting that 1024 was the system default and explicitly setting it to match, probably out of habit or a desire to be explicit. The comment was accurate when written. The machine did cap there.

The machine had not capped there for approximately fifteen years.

The archaeology

This is what config propagation actually looks like in production systems that have been running a while. You don’t inherit bugs — you inherit decisions that were correct under conditions that no longer exist, carrying no record of what conditions they were correct under. The 2003 author wrote a comment. The comment said machines cap here anyway. By 2008 that was already no longer true, but the migration engineer copied the script and didn’t read the comment carefully enough to notice that the rationale had expired. Or read it and trusted it. Either way, by the time I was looking at it in 2026, the comment was an artifact of a different world and nobody had the context to question it.

I spent a morning tracing this because it bothered me more than the incident itself. The incident was trivial. A ulimit bump, a restart, done. But the incident was trivially preventable and had been preventable for over fifteen years, waiting patiently for us to scale past 1024 concurrent connections, which we had finally done.

What bothered me: nobody knew the number was wrong because nobody knew the number existed. It was set in a script that was called by a supervisor that was configured by an ansible role that was documented in a README that nobody had read recently because the thing it described had been working fine. The thing was doing its job — starting the process — without doing the hidden job of starting it with the right limits. The difference is invisible until it isn’t.

What you don’t question

There’s a specific category of configuration value that is never questioned: the value that has always been there. Not the value that someone set recently and you can ask about. The value that predates institutional memory, that shows up in the config the same way background radiation shows up in a detector — always present, origin unclear, assumed natural.

1024 is a particularly good one because it’s a power of two and therefore looks like a deliberate technical choice. It has the aesthetic of intentionality. If the number were, say, 847, someone would have asked by now. 1024 reads like a sysadmin made a decision. The decision is plausible. The number persists.

I’ve seen this with other numbers: 8192 as a socket backlog because that’s what was in the example. 4096 as a stack size because someone read it in a book. 512 as a connection pool size because the person who set it was thinking about something else at the time and it was a round number. These values get copied faithfully through every migration because they’re not obviously wrong and there’s always a hundred things that are more obviously wrong to deal with first.

The technical word for this is cargo culting, but that’s a bit uncharitable to the people doing the copying — they’re not performing a ritual, they’re triaging, and the value that’s always been there and hasn’t caused problems is not the one you audit during a migration. The audit queue is full of things that are actively broken.

The fix, beyond the obvious

We bumped the limit. 65536 is the conventional choice: high enough to not matter, low enough to not panic the kernel, standard enough that you won’t confuse whoever looks at it next. We set it in three places — the process supervisor unit, /etc/security/limits.conf, and the systemd slice, because in a sufficiently complex system, the limit you think is set is not always the limit that applies. Belt, suspenders, and also don’t trust the pants.

We deleted the wrapper startup script. It had been doing one thing: calling ulimit -n 1024 before exec. Without that thing, it was just an exec wrapper for a binary that didn’t need wrapping. The script had been cargo-culted through three migrations. It was load-bearing in the sense that removing it required someone to double-check that nothing else was using it. Nothing else was.

We added the effective FD limit to the service’s startup log output. One line: pid=%d fd_limit_soft=%d fd_limit_hard=%d. Now every deploy produces a log entry with the actual limits in effect, which means the next person who cares can check the log instead of running lsof -p $(pgrep) | wc -l at 2am. It costs nothing.

We also audited every service in the cluster for the same pattern — not specifically for ulimit -n 1024, but for any limit set in a legacy wrapper script that postdated the current process supervisor config. Found three more. Two were harmless (limits well above actual usage). One was another 1024.

The thing worth saying

The incident postmortem listed the root cause as file descriptor limit set too low. That is accurate and also incomplete. The root cause is that we had no way to know what limits were actually in effect for running processes without manually checking each one, and we had no convention for surfacing limits in observable output, and we had no process for reviewing inherited config values during migrations.

The ulimit -n 1024 was a symptom. The disease is that configuration values carry forward without their rationale, and rationale is the only thing that lets you evaluate whether a value is still correct. Strip the rationale — by not writing it, by losing the author, by running the migration at 11pm on a deadline — and what’s left is a number that looks intentional, in a script that’s always been there, doing something that only becomes visible when it breaks.

Someone in 2003 wrote a comment explaining why 1024. That comment survived twenty-three years. The conditions it described did not. The comment was the only thing standing between us and this exact incident, and by the time we were in the incident, nobody had read it.

Write the rationale. When you migrate, read the rationale. If you can’t find the rationale for a limit, that’s the limit you audit. The one with no reason is the one with a reason that expired.

1024 should have been enough. It was, for a while. Then it wasn’t, and we didn’t know because the comment that explained the original reasoning had become a fossil — present, legible, and wrong.