preadv(2) and pwritev(2) are fine. Scatter/gather at a given offset, no fuss. But they inherited read(2)’s original sin: the only way to change I/O behavior is to change the fd’s flags with fcntl(F_SETFL), which is global state that affects every thread using that fd simultaneously.
Linux 4.6 added preadv2(2) and pwritev2(2) to fix that. The signatures are identical except for a fifth argument, flags, that applies only to that one call.
#include <sys/uio.h>
ssize_t preadv2(int fd, const struct iovec *iov, int iovcnt,
off_t offset, int flags);
ssize_t pwritev2(int fd, const struct iovec *iov, int iovcnt,
off_t offset, int flags);Pass offset as -1 to use the current file position, same as regular read(2). Pass 0 as flags and you have preadv(2) with zero behavior change. The interest is in the non-zero flags.
RWF_NOWAIT: the one you actually want
Added Linux 4.14, this is the flag with the widest real-world use. If the read would block waiting for data to arrive from disk (page not in cache, O_DIRECT on a slow device, lock contention), the kernel returns EAGAIN immediately instead of parking the thread.
This is how you build a fast path without io_uring. Read from cache if it is hot; fall back to a blocking read or queue a real async I/O if it is not.
struct iovec iov = { .iov_base = buf, .iov_len = len };
ssize_t n = preadv2(fd, &iov, 1, offset, RWF_NOWAIT);
if (n < 0 && errno == EAGAIN) {
/* page not hot; queue a blocking read or io_uring sqe */
}Constraints: RWF_NOWAIT only works on regular files and, as of Linux 5.14, on some block devices. Sockets, pipes, and character devices return EOPNOTSUPP. It also does not help if the fd itself has a lock held by another writer; that blocks regardless. And the obvious corollary: a cache miss in a read() loop that assumed cache-hot data and never handled EAGAIN will infinite-loop.
RWF_HIPRI: polled delivery
RWF_HIPRI tells the block layer to use polled I/O rather than interrupt-driven completion. The kernel spins waiting for the NVMe completion queue entry instead of sleeping and waking on interrupt. Reduces latency at the cost of CPU.
This only matters at tail latency on NVMe devices configured with polled queues (/sys/block/nvme0n1/queue/io_poll). On a spinning disk, an SSD without poll support, or a filesystem on top of device mapper, it degrades to a hint the kernel ignores. You will not get an error; you will get normal interrupt-driven I/O with no warning.
Combine with O_DIRECT to bypass the page cache entirely. Without O_DIRECT, the kernel can satisfy the read from cache and the poll path never fires.
RWF_DSYNC and RWF_SYNC: durability without opening O_DSYNC
If you have a shared fd and need one specific write to land on stable storage before you return, opening O_DSYNC forces every write on that fd to wait. RWF_DSYNC applies the same behavior to a single pwritev2(2) call. Data is written and the necessary metadata to recover it is on stable storage before the call returns. Equivalent to fdatasync(2) after the write, without the extra syscall, and only for this write.
RWF_SYNC is the stricter version: all metadata is synced, not just recovery metadata. Equivalent to fsync(2) after the write.
Neither flag is useful on O_DIRECT fds; the page cache is already bypassed and you own the sync semantics.
RWF_APPEND: atomic append per-call
Added Linux 4.16. Makes the write behave as O_APPEND regardless of the fd’s flags, and makes the supplied offset argument irrelevant (the write goes to the end). The atomicity guarantee is the same as O_APPEND: the seek-to-end and write are atomic with respect to other O_APPEND or RWF_APPEND writers on the same file.
The narrow use case is a shared fd without O_APPEND where you occasionally need one append without changing the fd flags globally. Append-only audit logs that share an fd with normal-offset reads are the common example.
FreeBSD
No preadv2/pwritev2. FreeBSD has preadv/pwritev since FreeBSD 6 (2005) and that is where the family stops. For polled NVMe, FreeBSD handles this in the GEOM layer and via cam_iosched, not via per-call fd flags. For durability, you call fdatasync(2) or fsync(2) explicitly.
Version and availability
preadv2/pwritev2: Linux 4.6 (2016), glibc 2.26. RWF_NOWAIT: Linux 4.14. RWF_APPEND: Linux 4.16. Include <sys/uio.h> for the prototype and <linux/fs.h> for the RWF_* constants if your glibc predates 2.26 and you fall back to syscall(2). No POSIX equivalent; no BSD equivalent.
For the kernel-side move after you have data in a pipe, see vmsplice and the file descriptors and IPC hub for the full scatter/gather family.