SO_REUSEPORT

Synopsis

#include <sys/socket.h>

int setsockopt(int sockfd, int level, int optname,
               const void optval[optlen], socklen_t optlen);
/* level = SOL_SOCKET, optname = SO_REUSEPORT, set before bind(2) */

Rod's take

One listening socket shared by every worker means one accept queue, one lock, and the whole herd waking up for each connection. SO_REUSEPORT gives each worker its own socket on the same port, and the kernel hashes connections across the group. Set it on every socket before bind, same effective UID on all of them. Don't like the hash? Attach a BPF program and pick the socket yourself. What it won't save you from: close a socket and the connections already parked in its accept queue go down with it. Drain before you restart.

Flags and arguments that matter

Errors you will actually hit

Example

int fd = socket(AF_INET6, SOCK_STREAM | SOCK_CLOEXEC, 0);
if (fd < 0)
    err(1, "socket");

int one = 1;
if (setsockopt(fd, SOL_SOCKET, SO_REUSEPORT, &one, sizeof(one)) < 0)
    err(1, "SO_REUSEPORT");
if (bind(fd, (struct sockaddr *)&addr, sizeof(addr)) < 0)
    err(1, "bind");
if (listen(fd, 1024) < 0)
    err(1, "listen");
/* run this in every worker; each accept()s on its own fd */

Trimmed from the full walkthrough in SO_REUSEPORT: each worker gets its own accept queue, and then the migration problem hits.

BSD equivalent

FreeBSD: SO_REUSEPORT_LB load-balances TCP and UDP across up to 256 sockets per group. OpenBSD: SO_REUSEPORT allows duplicate bindings, documented for multicast/broadcast reception rather than load balancing.