SO_REUSEPORT: each worker gets its own accept queue, and then the migration problem hits

Before Linux 3.9, a multi-process TCP server meant one socket shared across all workers. You call socket(), bind(), listen(), then fork. Every worker blocks on accept() against the same socket. A connection arrives, the kernel wakes every sleeping worker, one wins the race, the rest go back to sleep. This is the thundering herd problem and it is as old as Unix. The kernel worked around it in 3.x with SOCK_NONBLOCK and edge-triggered epoll on the shared socket, which prevented the useless wakeups. It did not fix the underlying serialization: the accept queue is still protected by a single socket lock, and every accept call contends on it.

The other option was accepting in one dedicated thread and handing connections off to workers. This moves the bottleneck from the kernel lock to your dispatch code, which is only marginally better and now you own the bug.

SO_REUSEPORT takes the other approach: give each worker its own socket. Each has its own accept queue, its own lock, no contention between workers. The kernel hashes incoming connections across the group and each worker calls accept() against only its own fd.

The socket setup

Set SO_REUSEPORT on every socket in the group before bind(). That is the entire API:

#include <sys/socket.h>
#include <netinet/in.h>

int make_reuseport_listener(const struct sockaddr *addr, socklen_t addrlen)
{
    int fd = socket(addr->sa_family, SOCK_STREAM | SOCK_CLOEXEC, 0);
    if (fd < 0)
        return -1;

    int one = 1;
    if (setsockopt(fd, SOL_SOCKET, SO_REUSEPORT, &one, sizeof(one)) < 0)
        goto err;

    if (bind(fd, addr, addrlen) < 0 || listen(fd, 1024) < 0)
        goto err;

    return fd;
err:
    close(fd);
    return -1;
}

Call this in each worker after fork, or in each thread before entering the accept loop. The kernel groups sockets by address:port. For TCP, incoming connections are distributed using a 4-tuple hash: source IP, source port, destination IP, destination port. The same 4-tuple always maps to the same socket in the group, so a client’s retransmits or retries reach the same worker, which matters if you are holding per-connection state.

UDP works the same way: datagrams hash to one socket, not broadcast to all of them.

The BPF dispatch

The 4-tuple hash is a fixed kernel policy. SO_ATTACH_REUSEPORT_EBPF lets you replace it: UDP support landed in Linux 4.5, TCP in 4.6.

Attach an eBPF program to any socket in the group. The kernel runs it for every incoming connection instead of the hash. Two program types are accepted. The original is BPF_PROG_TYPE_SOCKET_FILTER. The one you actually want, since Linux 4.19, is BPF_PROG_TYPE_SK_REUSEPORT, and its context is struct sk_reuseport_md:

struct sk_reuseport_md {
    void    *data;         /* start of packet headers */
    void    *data_end;
    __u32    len;          /* packet length */
    __u32    hash;         /* kernel's own hash, if you want it */
    __u32    reuseport_id; /* stable id for the socket group */
    struct bpf_sock *migrating_sk; /* non-NULL during a migration (Linux 5.14+) */
    /* trimmed: eth_protocol, ip_protocol, bind_inany, sk */
};

The two types return different things, and mixing them up is how you lose an afternoon. A SOCKET_FILTER program returns an index between 0 and N-1, the socket’s position in the group; an invalid index falls back to the plain hash. An SK_REUSEPORT program returns an action, SK_PASS or SK_DROP, and picks the socket by calling bpf_sk_select_reuseport() before it returns. Return SK_PASS without selecting anything and the kernel falls back to its own hash.

The common use is session or tenant affinity. Parse the first few bytes of packet data, extract an identifier, and map it to a specific worker index. You can also override the kernel hash with something stable across worker restarts, which matters for the migration problem below.

SO_ATTACH_REUSEPORT_CBPF takes classic BPF instead of eBPF, with the same semantics and less capability. Either works for simple dispatch.

BPF_MAP_TYPE_REUSEPORT_SOCKARRAY

Index-based dispatch breaks when the group size changes. If you have four workers at indices [0, 1, 2, 3] and worker 1 exits, the group shrinks to three sockets and the indices shift. Any session affinity keyed to fixed slot numbers now routes some sessions to the wrong worker.

BPF_MAP_TYPE_REUSEPORT_SOCKARRAY (Linux 4.19) solves this by storing actual socket file descriptors in a BPF map. The dispatch program selects by map key instead of by position:

struct {
    __uint(type, BPF_MAP_TYPE_REUSEPORT_SOCKARRAY);
    __uint(max_entries, 256);
    __type(key, __u32);
    __type(value, __u64);
} worker_map SEC(".maps");

SEC("sk_reuseport")
int dispatch(struct sk_reuseport_md *ctx)
{
    __u32 key = ctx->hash % 256;
    /* 0 on success, -errno if the slot is empty. Either way, pass:
     * no selection means the kernel falls back to its own hash.
     * Return SK_DROP instead if you'd rather fail closed. */
    bpf_sk_select_reuseport(ctx, &worker_map, &key, 0);
    return SK_PASS;
}

Do not return bpf_sk_select_reuseport(...) directly. The helper returns 0 on success, and SK_DROP is 0. Congratulations, you have written a program that drops every connection it successfully routes.

When a worker restarts, update the map entry for its slot to point at the new socket before removing the old one. The BPF program always routes by map lookup, so there is no index shift and no window where a session key maps to nowhere.

The migration problem

None of the above touches the hard case: a socket is removed from the reuseport group while connections are in flight.

When a listening socket closes, everything still tied to it dies with it: requests in SYN_RECV (three-way handshake not yet complete) and connections fully established and sitting in the accept queue waiting for a worker to call accept(). The kernel bound each one to that specific listener when the SYN arrived, and by default it does not hand them to another socket in the group. The client gets a RST or a timeout.

For rolling restarts, the practical options are:

Drain before closing. Get the new socket listening and added to the BPF map before removing the old socket from it. Stop routing new connections to the old socket by updating the map, wait long enough for any in-flight SYNs to have either completed or timed out (a few hundred milliseconds is conservative on a local network, longer on WAN), then close. The accept queue should be empty by then because you stopped accepting new connections and the worker was draining it. This requires that your BPF dispatch program actually respects the map and that you control the map update timing.

Linux 5.14 added listener migration. Set net.ipv4.tcp_migrate_req=1 and, when a listener is closed or shut down, its in-flight handshakes and the established connections still sitting in its accept queue get moved to another listener in the group instead of being reset. Without a program the kernel picks a live listener at random; with an SK_REUSEPORT program attached as BPF_SK_REUSEPORT_SELECT_OR_MIGRATE, your program gets called with migrating_sk set (NULL on a normal lookup) and chooses, or returns SK_DROP to cancel the migration. Do that when the listeners aren’t configured identically: the kernel docs’ example is a request migrating to a listener with TCP_SAVE_SYN that never saved the SYN. The sysctl defaults to 0, so on a stock box nothing migrates and the drain approach above is still what saves you. Data already read off an accepted socket is your process’s problem, not the kernel’s.

Contention on the socket lock

SO_REUSEPORT eliminates cross-worker contention. It does not eliminate contention within a single worker’s socket. If one worker is accepting connections faster than it processes them, its accept queue lock is still a serialization point. The queue fills, the kernel starts dropping or deferring SYNs. Tune net.core.somaxconn and net.ipv4.tcp_max_syn_backlog appropriately. The Recv-Q column in ss -tlnp shows the current accept queue depth for each listening socket; a queue that is perpetually at or near its limit is the diagnostic, not a guess. The ss internals tip covers what those columns actually mean. If a single worker’s queue is chronically full, you need more workers or faster processing, not more tuning.

FreeBSD

FreeBSD had SO_REUSEPORT since 4.4BSD with different semantics: multiple sockets can bind to the same address:port, but UDP datagrams are replicated to all sockets in the group rather than distributed. This is useful for multicast receivers. It is not useful for multi-worker load distribution.

FreeBSD 12.0 added SO_REUSEPORT_LB, which provides Linux-style distribution: connections and datagrams hash to one socket, not all. Write code that should run on both systems conditionally:

#ifdef __linux__
    setsockopt(fd, SOL_SOCKET, SO_REUSEPORT, &one, sizeof(one));
#elif defined(__FreeBSD__)
    setsockopt(fd, SOL_SOCKET, SO_REUSEPORT_LB, &one, sizeof(one));
#endif

The eBPF dispatch (SO_ATTACH_REUSEPORT_EBPF) is Linux-only. FreeBSD’s BPF layer does not have an equivalent hook at the socket group level.

---

The SCM_RIGHTS tip covers the complementary pattern: accepting in one place and passing the live fd to another process without any of the reuseport machinery. Different trade-off, worth knowing both.

The kernel implementation of reuseport dispatch is in net/core/sock_reuseport.c. It is short and readable. The accept queue mechanics that underlie all of this are in W. Richard Stevens’ TCP/IP Illustrated Vol. 1 (Amazon affiliate link; RodHat may earn a commission) in the TCP connection establishment chapters; understanding what is actually in that queue makes the migration problem obvious instead of surprising.