The tuning lasted until the reboot.

The box was a high-traffic nginx reverse proxy. Sixteen cores, 64GB of RAM, and a listen queue that spent most of its life at 128 because nobody had looked at net.core.somaxconn since the server was commissioned in 2022. The symptoms were intermittent connection resets under load, the kind that show up in client logs as ECONNREFUSED and in your on-call rotation as “the site is flaky.”

We spent most of a day on it. Found the listen queue, the same sort of blind spot as the service that was healthy but accepted no new connections. Found vm.swappiness at 60 on a machine that should never be swapping. Found net.core.netdev_max_backlog at 1000 when the NIC could saturate it in milliseconds. Tuned all of it. Ran benchmarks before and after. The connection resets went away. We wrote up the incident and closed the ticket.

Six months later, the same symptoms were back.

---

It took about two hours of investigation to notice the obvious.

$ sysctl net.core.somaxconn
net.core.somaxconn = 128

128. We had set this to 4096. I had set it to 4096. I watched the command run. It printed the new value. I saw it work.

$ grep somaxconn /etc/sysctl.conf /etc/sysctl.d/* 2>/dev/null

Nothing.

$ grep -r somaxconn /etc/ 2>/dev/null

Nothing. The value lived nowhere on disk. It had existed only in the running kernel, from the moment we set it in April until the moment the maintenance reboot fired in October.

sysctl -w succeeded. It just did not persist. There is no warning. The command prints the new value and exits zero and goes home, and it is your problem to understand the difference between “write to the running kernel” and “configure the system.”

---

Here is the thing about /proc/sys/. It is a filesystem that represents running kernel state. When you write to /proc/sys/net/core/somaxconn or call sysctl -w net.core.somaxconn=4096, you are writing to that filesystem. The kernel reads it, updates the in-memory value, and the change takes effect immediately. It does not write to disk, because /proc/sys/ is not on disk. It is in memory. The kernel creates it fresh every boot from compiled-in defaults plus whatever /etc/sysctl.d/ tells it to load.

The write succeeds. The read confirms it. From the kernel’s perspective, the contract is fulfilled. You asked for a value, you got a value. Nobody promised it would survive power loss.

This is not a bug. It is the documented behavior, right there in sysctl(8) if you read past the examples. The -w flag is for runtime adjustment. The /etc/sysctl.d/ files are for configuration. These are different things with different lifetimes and the tool makes no effort to bridge them for you.

I know this. I have known this for twenty years. And in April, under pressure, with a production outage in progress, I ran sysctl -w on six values, verified that they worked, and closed the ticket without writing a single one of them to disk.

---

The fix is not complicated.

Create a file in /etc/sysctl.d/. Not /etc/sysctl.conf directly. On modern Debian and Ubuntu, the package manager owns /etc/sysctl.conf and will clobber it on updates. Use a numbered drop-in under /etc/sysctl.d/ and it will survive both package upgrades and reboots.

$ cat /etc/sysctl.d/99-rodhat-tuning.conf
# nginx listen queue and network buffer tuning
net.core.somaxconn = 4096
net.core.netdev_max_backlog = 16384
net.ipv4.tcp_max_syn_backlog = 4096
vm.swappiness = 10

Apply it to the running kernel without a reboot:

$ sysctl -p /etc/sysctl.d/99-rodhat-tuning.conf
net.core.somaxconn = 4096
net.core.netdev_max_backlog = 16384
net.ipv4.tcp_max_syn_backlog = 4096
vm.swappiness = 10

Now reboot. Check. The values are still there. File and running kernel agree, because the boot sequence reads the drop-in and applies it before nginx starts.

The 99- prefix matters. The files load in lexicographic order. Your overrides need to come after the distro defaults in 10-network.conf and 10-console.conf and whatever else your distribution ships. A 99- prefix puts you last and you win.

---

After fixing the immediate problem, I wrote a small verification script that we added to the post-maintenance-window runbook:

#!/bin/sh
# Verify sysctl values match their /etc/sysctl.d configuration.
# Run after every reboot to catch unmanaged runtime tuning.

MISMATCHES=0

for conf in /etc/sysctl.d/*.conf /etc/sysctl.conf; do
    [ -f "$conf" ] || continue
    while IFS='= ' read -r key value; do
        [ -z "$key" ] && continue
        case "$key" in \#*) continue ;; esac
        live=$(sysctl -n "$key" 2>/dev/null)
        if [ "$live" != "$value" ]; then
            printf 'MISMATCH: %s: want %s, got %s\n' "$key" "$value" "$live"
            MISMATCHES=$(( MISMATCHES + 1 ))
        fi
    done < "$conf"
done

[ "$MISMATCHES" -eq 0 ] && echo "All sysctl values match configuration." || exit 1

Nothing fancy. Reads the configured values, compares them to the running kernel, shouts if they differ. Takes two seconds. We have run it after every reboot since October and it has caught two more cases where someone ran sysctl -w on a different box and never wrote it down.

---

There is a broader category of failure that lives here. The pattern is: a tool accepts writes at runtime, reports success, and has a completely separate mechanism for making those writes durable. The runtime interface and the persistence interface look similar enough that under pressure you use the wrong one and do not notice until something reboots.

iptables -A is in this category. The rule is active immediately. iptables-save and iptables-restore are how you make it survive a reboot. If you use ufw or firewalld or nftables with a proper service unit, this is handled for you. If you are calling iptables by hand, you are in sysctl -w territory.

ip route add is in this category. The route works until the network service restarts or the box reboots.

modprobe is in this category. The module loads now. If you need it at boot, it goes in /etc/modules or a .conf file under /etc/modules-load.d/.

The common thread: the runtime interface is intentionally transient, because transient is what you want when you are diagnosing a problem in production and you need to try a value without committing to it. The problem is that the same interface is also what you reach for when you are in a hurry and just want the thing to work, and the transience feels like a feature until six months later. It belongs on the same reliability and observability shelf as every other fix that looked done until the next restart.

The rule I came away with: if you use sysctl -w on a production box, you have thirty seconds of work left. If you skip those thirty seconds, you have scheduled an outage for sometime in the next twelve months, whenever the box next reboots. Usually during a maintenance window. Sometimes at three in the morning when a kernel panic takes the box down and it comes back up with defaults.

Write it to disk before you close the ticket. The command has already done the hard part.