$RodHat_
Console Tips

splice and tee: zero-copy data movement, 2.6.17 edition

Published by

splice and tee: zero-copy data movement, 2.6.17 edition
Photo: AI-generated — no human photographer / RodHat AI Cover

The read/write loop has been embarrassing C programmers since Version 7 Unix. You open a file, read a chunk into userspace, write the same chunk to a socket or another file. The data makes two unnecessary round trips through your process address space: once into a buffer you allocated, once out of it. The kernel copies from page cache into your buffer. You copy from your buffer back into kernel space. The page cache already had the data. You just put it somewhere else and handed it back.

sendfile(2) arrived in Linux 2.2 to fix the file-to-socket case. It works, but it is a special case: file in, socket out, done. You cannot chain it.

splice(2), tee(2), and vmsplice(2) landed in 2.6.17 (June 2006) and do the general case. Twenty years in the kernel. Still underused because almost nobody reads man pages anymore.

What splice does

The pipe is the key. A pipe in Linux is a ring buffer of kernel pages. When you write() to a pipe, the kernel does not copy your bytes into the pipe’s internal buffer for large writes. It hands page references over. When you read() from the pipe, same deal in reverse. The actual bytes stay put in the page cache; only ownership metadata moves.

splice(2) generalizes this:

#define _GNU_SOURCE
#include <fcntl.h>

ssize_t splice(int fd_in, loff_t *off_in,
               int fd_out, loff_t *off_out,
               size_t len, unsigned int flags);

One end must always be a pipe. Both ends can be pipes. The flags worth knowing:

  • SPLICE_F_MOVE: hint to move pages rather than copy them (advisory, kernel can ignore it)
  • SPLICE_F_NONBLOCK: return EAGAIN rather than blocking on a full or empty pipe
  • SPLICE_F_MORE: more data is coming, analogous to MSG_MORE for TCP corking

File to socket, done right

The classic case. You want to send a file to a network socket without touching userspace:

int copy_file_to_socket(int file_fd, int sock_fd, off_t offset, size_t size)
{
    int pfd[2];
    if (pipe(pfd) < 0) return -1;

    ssize_t total = 0;
    while ((size_t)total < size) {
        ssize_t n = splice(file_fd, &offset, pfd[1], NULL,
                           size - (size_t)total,
                           SPLICE_F_MOVE | SPLICE_F_MORE);
        if (n <= 0) break;

        ssize_t sent = splice(pfd[0], NULL, sock_fd, NULL,
                              (size_t)n, SPLICE_F_MOVE | SPLICE_F_MORE);
        if (sent <= 0) break;
        total += sent;
    }

    close(pfd[0]);
    close(pfd[1]);
    return (int)total;
}

Two calls. The first splice moves the file’s page cache pages into the pipe. The second moves them from the pipe to the socket. No userspace copy. The path is: page cache to network card DMA, with only page table entries shuffled in between.

sendfile(2) is a kernel shortcut for exactly this pattern with a regular file as input and a socket as output, hiding the intermediate pipe. Under the hood, on many code paths, it calls splice. If sendfile covers your use case, use it; the interface is simpler. When you need composability, a non-socket output, or more control, splice is the primitive.

tee: fanout without a copy

tee(2) duplicates data from one pipe to another without consuming the source:

ssize_t tee(int fd_in, int fd_out, size_t len, unsigned int flags);

Both fds must be pipes. The data in fd_in is not consumed. A subsequent read() or splice() on fd_in sees the same data again. This is how you do log fanout without spawning a tee(1) subprocess.

Say you have a long-running process writing to a pipe. You want that output going to a log file AND to a monitoring socket, in real time, without forking a shell pipeline:

/*
 * Forward all data from src_r to both dest_a (a file fd)
 * and dest_b (a monitoring socket), without copying through userspace.
 * bridge is a scratch pipe for the second consumer.
 */
void fanout_loop(int src_r, int dest_a, int dest_b)
{
    int bridge[2];
    pipe(bridge);

    for (;;) {
        /* duplicate: src still has the data, bridge gets a copy */
        ssize_t n = tee(src_r, bridge[1], 65536, SPLICE_F_NONBLOCK);
        if (n < 0 && errno == EAGAIN) {
            /* nothing available yet; poll src_r and try again */
            continue;
        }
        if (n <= 0) break;

        /* consume original data from src, send to dest_a */
        splice(src_r, NULL, dest_a, NULL, (size_t)n, SPLICE_F_MOVE);

        /* consume bridge copy, send to dest_b */
        splice(bridge[0], NULL, dest_b, NULL, (size_t)n, SPLICE_F_MOVE);
    }

    close(bridge[0]);
    close(bridge[1]);
}

The shell tee(1) command does the same thing conceptually: read from stdin, write to stdout and a file. The difference is it goes through userspace. The kernel tee(2) syscall stays in the page cache.

vmsplice: going the other direction

vmsplice(2) pushes userspace pages into a pipe:

#include <sys/uio.h>

ssize_t vmsplice(int fd, const struct iovec *iov,
                 size_t nr_segs, unsigned int flags);

fd must be the write end of a pipe. iov describes your userspace buffers. With SPLICE_F_GIFT set, you hand those pages to the kernel outright and must not touch them again. Without it, the kernel copies them, which costs a copy but lets you reuse the buffer immediately after the call returns.

The pattern: allocate page-aligned buffers, produce data into them, vmsplice into a pipe, then splice from the pipe to a socket or file:

void send_buffer_to_socket(int sock_fd, void *buf,
                            size_t len, int pfd[2])
{
    struct iovec iov = { .iov_base = buf, .iov_len = len };

    /* move buf into pipe (kernel copies once without SPLICE_F_GIFT) */
    vmsplice(pfd[1], &iov, 1, 0);

    /* move pipe to socket (no userspace copy from here) */
    splice(pfd[0], NULL, sock_fd, NULL, len, SPLICE_F_MOVE);
}

If your producer generates data in userspace before shipping it, vmsplice is as close to zero-copy as you get without writing a kernel module.

The constraint: one end is always a pipe

Every time, no exceptions. You cannot splice directly from a file to another file. You cannot splice from one socket to another. The pipe is the kernel’s staging buffer.

If you need file-to-file zero-copy, you splice through a scratch pipe, same as the network example above. If you are building a transparent proxy doing socket-to-socket forwarding, you splice in and then splice out, pipe in the middle. This is not an accident. The pipe’s buffer is what makes the no-userspace-copy guarantee possible; it gives the kernel a stable place to hold page references while data moves between fds.

Pipe capacity

Default pipe capacity on modern Linux is 65536 bytes, 16 pages of 4KB. For high-throughput paths, bump it before splicing:

/* raise pipe capacity to 1MB */
fcntl(pfd[1], F_SETPIPE_SZ, 1 << 20);

The ceiling is /proc/sys/fs/pipe-max-size, default 1MB on most distributions. Root can raise it. Unprivileged processes are capped at whatever the sysctl says.

When the pipe is smaller than the requested len, splice returns early with fewer bytes than requested. Loop on short returns the same as you would on a short write().

When to use io_uring instead

io_uring has io_uring_prep_splice(), which composes splice-equivalent transfers with batched syscalls and completion rings. If your program is already io_uring-native for async I/O, stay there and use the splice prep op.

If you are writing a straightforward synchronous service that just needs to forward data efficiently, splice and tee are less overhead to wire up and have twenty years of production soak. They do not require ring setup, SQE/CQE management, or anything else. Pick the simpler tool when the simpler tool is sufficient.

Observing the difference with bpftrace

If you suspect a data path is doing unnecessary userspace copies, bpftrace can confirm it:

# count copy_from_user calls per process for 10 seconds
bpftrace -e 'kfunc:copy_from_user { @[comm] = count(); } interval:s:10 { print(@); clear(@); exit(); }'

High copy_from_user counts on a process doing bulk I/O means it is routing data through userspace buffers on reads. Switching that path to splice or sendfile should drop the count toward zero for the bulk-transfer portion.

Composing with memfd_create

The memfd_create post covers anonymous file descriptors that slot into splice pipelines cleanly. A memfd_create fd is seekable and can be ftruncated like a real file, but lives entirely in RAM with no filesystem path. You can splice from a memfd into a pipe the same way you would from a file on disk. Useful when a producer generates data in memory and you want seekable access before sending it over the wire: write to memfd, seek to zero, splice to pipe, splice to socket.

splice and tee predate every modern zero-copy framework by a decade or more. Moving kernel page references instead of data bytes is not a new idea. The man page has been there the whole time. If you are reading data into a userspace buffer because that is what the template said to do, you have a twenty-year-old bug.