$RodHat_
Console Tips

SCM_RIGHTS: send an open file descriptor to another process over a Unix socket

Published by

SCM_RIGHTS: send an open file descriptor to another process over a Unix socket
Photo: AI-generated — no human photographer / RodHat AI Cover

You opened a file before dropping privileges. The file is gone from the filesystem now, or the path is meaningless inside a chroot, or you are in a separate network namespace and the socket is already bound and listening and you want a worker process to have it. Your first instinct is probably /proc/<pid>/fd/<n>. That is the wrong instinct.

The right mechanism is SCM_RIGHTS. It has been in POSIX since before Linux was born, it works on FreeBSD and Linux and every sane Unix, and it passes the live kernel file description: same offset, same flags, same permissions, reference count bumped by one. The receiver gets a new file descriptor number in its own table pointing at the same kernel object.

File descriptor vs. file description

This distinction matters here. A file descriptor is a small integer in your process’s fd table. A file description is the kernel object the fd refers to: it holds O_RDWR/O_APPEND/etc., the current seek offset, and the inode reference.

dup(2) creates two file descriptors pointing at one description. Change the offset through one and both see it. fork(2) copies the fd table: the child gets different descriptor numbers that point at the same descriptions. SCM_RIGHTS is the same idea across process boundaries: the receiver gets a new number, the underlying object is shared, the kernel keeps it alive until both sides close their handles.

The mechanism

Unix domain sockets support ancillary data: out-of-band information that travels alongside the byte stream. sendmsg(2) and recvmsg(2) access it via msg_control in struct msghdr. SCM_RIGHTS is one ancillary data type. You pack file descriptors into a control message, call sendmsg, and the kernel handles the cross-process transplant atomically.

Sender:

#define _GNU_SOURCE
#include <sys/socket.h>
#include <string.h>

int send_fd(int sockfd, int fd_to_send)
{
    struct msghdr msg  = {};
    struct iovec  iov;
    struct cmsghdr *cmsg;
    char dummy         = '\0';
    char buf[CMSG_SPACE(sizeof(int))];

    /*
     * sendmsg requires at least one iov entry. The byte payload is
     * irrelevant here; we're here for the control message.
     */
    iov.iov_base       = &dummy;
    iov.iov_len        = sizeof(dummy);
    msg.msg_iov        = &iov;
    msg.msg_iovlen     = 1;

    msg.msg_control    = buf;
    msg.msg_controllen = sizeof(buf);

    cmsg               = CMSG_FIRSTHDR(&msg);
    cmsg->cmsg_level   = SOL_SOCKET;
    cmsg->cmsg_type    = SCM_RIGHTS;
    cmsg->cmsg_len     = CMSG_LEN(sizeof(int));
    memcpy(CMSG_DATA(cmsg), &fd_to_send, sizeof(int));
    msg.msg_controllen = cmsg->cmsg_len;

    return sendmsg(sockfd, &msg, 0);
}

Receiver:

int recv_fd(int sockfd)
{
    struct msghdr  msg  = {};
    struct iovec   iov;
    struct cmsghdr *cmsg;
    char dummy[1];
    char buf[CMSG_SPACE(sizeof(int))];
    int  received_fd    = -1;

    iov.iov_base       = dummy;
    iov.iov_len        = sizeof(dummy);
    msg.msg_iov        = &iov;
    msg.msg_iovlen     = 1;

    msg.msg_control    = buf;
    msg.msg_controllen = sizeof(buf);

    if (recvmsg(sockfd, &msg, MSG_CMSG_CLOEXEC) < 0)
        return -1;

    if (msg.msg_flags & MSG_CTRUNC)
        return -1; /* kernel dropped the fd because our buffer was too small */

    cmsg = CMSG_FIRSTHDR(&msg);
    if (cmsg
        && cmsg->cmsg_level == SOL_SOCKET
        && cmsg->cmsg_type  == SCM_RIGHTS)
        memcpy(&received_fd, CMSG_DATA(cmsg), sizeof(int));

    return received_fd;
}

Both ends need a connected AF_UNIX socket pair. Before the fork:

int sv[2];
socketpair(AF_UNIX, SOCK_STREAM, 0, sv);
/* parent uses sv[0], child uses sv[1]. close the unused end in each. */

MSG_CMSG_CLOEXEC

recvmsg on Linux 2.6.31+ accepts MSG_CMSG_CLOEXEC in the flags argument. When set, every fd received via SCM_RIGHTS in that call is marked O_CLOEXEC. You want this. Without it, any received fd is inherited by children your process forks later, which is almost certainly not what you intended. Set MSG_CMSG_CLOEXEC every time you receive ancillary data that contains fds.

MSG_CTRUNC is not your friend

If the receiver’s msg_controllen buffer is too small to hold the incoming control message, the kernel closes the excess file descriptors and sets MSG_CTRUNC in msg_flags. This is not graceful truncation you can recover from; the fds are gone. Check for it. Size your buffer correctly: CMSG_SPACE(sizeof(int) * n) for n file descriptors.

Why this beats the alternatives

/proc/<pid>/fd/<n> requires knowing the sender’s PID, having appropriate permissions (which tightened considerably in Linux 5.x), and tolerates a TOCTOU race if the sending process exits between your stat and your open. It also breaks entirely across PID namespaces.

Re-opening from the path: you get a new file description. New offset from zero. Whatever permissions the filesystem grants you at open time, which is not necessarily what the original opener had. If the file was deleted between the original open and your re-open, you are done. If you opened it before dropping privileges specifically to retain access you no longer have, re-opening defeats the entire purpose.

Forking and inheriting: works when the receiver is a child you are about to spawn, and you know at fork time exactly which fds to pass. Completely useless for passing fds to processes already running.

SCM_RIGHTS has none of those problems. The description arrives in the receiver’s table live, with no race, no path required, no privilege escalation needed.

Where this actually shows up

Privilege separation: the canonical use. A parent opens a raw socket or binds a port below 1024, passes the fd to an unprivileged worker over a socketpair, then drops root itself or exits. The worker gets the open socket without ever having held the privilege. OpenSSH’s privilege-separated architecture works on this principle. So does sshd on OpenBSD.

Socket activation: if someone tells you systemd invented socket activation, they have not read code written before 2010. The mechanism at the bottom is SCM_RIGHTS used at one remove: an init or supervisor pre-binds a listening socket, passes the fd to the daemon, and the daemon starts with LISTEN_FDS already pointing at fd 3 (or 3 through 3+n). You can implement the whole pattern without any service manager: write a small supervisor, bind the socket, fork the daemon, pass the fd over a socketpair before exec. The supervision-without-systemd tip is the starting point for that approach.

Container runtimes: the OCI runtime spec requires passing open terminal file descriptors into container namespaces. You cannot exec your way across namespace boundaries with just a path; the handle has to travel as a live fd. runc and containerd both do this via SCM_RIGHTS.

Database connection handoff: a privileged broker holds a pool of already-authenticated database connections. Worker processes request a connection; the broker passes the live socket fd. The worker inherits the authenticated session without ever seeing the credentials or touching the authentication round-trip.

Sending multiple fds at once

Scale CMSG_SPACE and CMSG_LEN to match:

int fds[3] = { fd0, fd1, fd2 };
char buf[CMSG_SPACE(sizeof(fds))];

cmsg->cmsg_len = CMSG_LEN(sizeof(fds));
memcpy(CMSG_DATA(cmsg), fds, sizeof(fds));

Linux’s hard limit is SCM_MAX_FD (255 in include/linux/unix.h). In practice, if you are sending more than a handful of file descriptors in one shot, the design probably needs examination before the fd count does.

Close what you receive

Every fd that arrives via SCM_RIGHTS is a real open file description. The kernel keeps it alive until all holders close it. Long-lived servers that loop on recvmsg and forget to close received fds will leak descriptions indefinitely. This is not a theoretical concern; it is a slow resource exhaustion that shows up as “open files” in lsof and nothing obvious in application logs. close(received_fd) when you are done with it, same as any other fd.

FreeBSD note

This is POSIX. The same SCM_RIGHTS, the same CMSG_* macros, compile on FreeBSD 14.x unchanged with _POSIX_C_SOURCE or _BSD_SOURCE. FreeBSD has used this for privilege-separated daemons longer than Linux has had the mechanism at all. The pidfd mechanism for process handles is Linux-specific; fd passing over Unix sockets is not.

The full ancillary data machinery, including SCM_CREDENTIALS and the complete CMSG_* macro set, is in chapter 61 of The Linux Programming Interface by Kerrisk. If you are doing anything serious with Unix sockets and you do not own that book, fix that before you read another blog post about it.