The kernel keyring has been in Linux since 2.6.10 and you have never used it
Published by RodHat · Updated
Environment variables are where secrets go to die. The value sits in /proc/PID/environ, readable by anything running as the same UID with permission on that file. It ends up in core dumps. On some ps implementations, ps e spits it into terminal output. Every subprocess you fork inherits it whether you intended that or not, and there is no revocation story: once the secret is in the environment it is there until the process exits.
Linux has had a better answer since kernel 2.6.10, released December 2004. The kernel keyring is a typed, permission-controlled, TTL-capable secret store that lives entirely in kernel memory. It handles fork and exec inheritance through a search hierarchy you configure per-key, not per-environment. Most production services have never touched it.
The five keyrings
Every thread has access to five standard keyrings. They form a search hierarchy:
Thread keyring (@t): private to the calling thread. Other threads in the process cannot see it.
Process keyring (@p): shared across all threads in a process. Not inherited: a fork() child starts with no process keyring of its own, and execve clears it. Destroyed when the last thread referring to it exits.
Session keyring (@s): inherited across fork and exec. PAM and SSH both create a session keyring on login and attach it to the session. Everything you spawn in that login session inherits it. When the session ends, the keyring and every key linked into it are destroyed.
User keyring (@u): tied to the UID. Outlasts individual sessions and is shared across all processes running as that UID. Useful for service accounts that restart between deployments.
User session keyring (@us): a fallback created automatically when a process has no session keyring. Common in cron jobs and early-boot services.
For a daemon that should lose its secret on restart, the process keyring is right. For a secret that should survive a daemon restart but still be scoped to the service account, the user keyring. For interactive sessions where the secret should expire on logout, the session keyring.
keyctl from the shell
The keyctl command from the keyutils package handles everything the syscall does, which is useful for scripting and debugging:
# Add a user key to the session keyring, prints the key ID
keyctl add user myapp_db_password "hunter2" @s
# 712345678
# List keys in the session keyring
keyctl list @s
# 1 key in keyring:
# 712345678: --alswrv 1000 1000 user: myapp_db_password
# Print the payload as raw bytes
keyctl pipe 712345678
# hunter2
# Set a TTL of 300 seconds
keyctl timeout 712345678 300
# Revoke: immediately invalidates, schedules garbage collection
keyctl revoke 712345678The TTL is real: the kernel garbage-collects expired keys without any userspace timer. Set it to the minimum useful lifetime of the secret. If your deploy process rotates credentials every 15 minutes, a 16-minute TTL turns a rotation miss into a hard failure rather than silent stale-credential drift.
The syscall layer
Three syscalls do the work: add_key(2), request_key(2), and keyctl(2).
#include <sys/types.h>
#include <keyutils.h> /* -lkeyutils */
/* Add a key of type "user" to the process keyring */
key_serial_t key = add_key("user",
"myapp:db:password",
password_buf, password_len,
KEY_SPEC_PROCESS_KEYRING);
if (key == -1) {
perror("add_key");
exit(1);
}
/* Set a 10-minute TTL */
if (keyctl(KEYCTL_SET_TIMEOUT, key, 600) < 0)
perror("KEYCTL_SET_TIMEOUT"); /* non-fatal, log and continue */KEY_SPEC_PROCESS_KEYRING is the constant for @p. The full set: KEY_SPEC_THREAD_KEYRING, KEY_SPEC_PROCESS_KEYRING, KEY_SPEC_SESSION_KEYRING, KEY_SPEC_USER_KEYRING, KEY_SPEC_USER_SESSION_KEYRING.
Retrieving the key in the same process (a child won’t find it in @p; see below):
/* Search the session, process, and thread keyrings */
key_serial_t key = request_key("user",
"myapp:db:password",
NULL, /* NULL callout: fail if missing */
0);
if (key == -1) {
perror("request_key");
exit(1);
}
void *payload = NULL;
long len = keyctl_read_alloc(key, &payload);
if (len < 0) {
perror("keyctl_read_alloc");
exit(1);
}
connect_db(payload, (size_t)len);
explicit_bzero(payload, (size_t)len);
free(payload);keyctl_read_alloc from libkeyutils handles the two-step read (call with NULL to get the length, then allocate and read) so you do not have to do it by hand. Use it instead of the raw keyctl(KEYCTL_READ, ...) loop.
The process keyring does not survive fork. The child gets a fresh, empty @p, and an exec wipes it again. That’s the point of @p: the secret dies with the process that loaded it. The session keyring is the one that’s inherited across fork, vfork, and execve. If the child needs the key, add it with KEY_SPEC_SESSION_KEYRING instead (ideally after keyctl_join_session_keyring(NULL) gives the service its own anonymous session, so you’re not stuffing it into your login session), and request_key finds it from the child without any extra wiring. No pipes, no getenv("DB_PASSWORD") in the child, no /proc/PID/environ exposure.
Key types
The type string matters more than most documentation makes clear.
user: standard type for application secrets. Payload is arbitrary bytes up to roughly 32KB. Readable by the owner via KEYCTL_READ. This is what you want.
logon: internally identical to user but KEYCTL_READ is refused from userspace. The payload can only be consumed by kernel code: filesystem drivers, PAM credential helpers, kerberos. If you store something as logon you cannot get it back from application code. Correct for kernel subsystems, wrong for a webserver.
keyring: a key whose payload is a list of other key IDs. This is the mechanism that builds the hierarchy. @s is itself a keyring. You can create sub-keyrings, link specific keys into them, and search hierarchically. Useful for scoping secrets to subsystems rather than dumping everything into the session keyring.
trusted: the payload is sealed by the TPM. The kernel generates a key inside the TPM, seals it against the current PCR state, and stores the sealed blob. If the PCR values have not changed since sealing (no tampered boot path), the TPM unseals and the key is available. If they have changed, decryption fails and the service fails loudly. Right for disk encryption keys and boot integrity proofs. Requires the trusted_keys kernel module and a TPM. Needs CAP_SYS_ADMIN to create, per the capabilities post.
encrypted: a key encrypted by another key (a trusted or user master key). The payload is stored encrypted; decryption happens in-kernel on access. Useful for key hierarchies where one TPM-sealed master unlocks many derived keys.
Key permissions
Keys have a four-group permission model: owner, group, other, and “possessor” (any process that has the key linked in one of its keyrings). Each group has independent bits for view, read, write, search, link, and setattr.
keyctl describe gives the full picture:
keyctl describe 712345678
# 712345678: alswrv;alswrv;;;;;; 1000 1000 user: myapp_db_passwordFor most service keys: owner reads, nobody else does anything. Set permissions with keyctl setperm. The permission value is a hex bitmask; owner-all is 0x3f000000:
keyctl setperm 712345678 0x3f000000The bit layout is in keyctl(2) and keyutils.h.
Quotas and /proc/keys
The kernel enforces per-UID quotas. Defaults:
cat /proc/sys/kernel/keys/maxkeys # 200 keys per user
cat /proc/sys/kernel/keys/maxbytes # 20000 bytes payload per userCheck current usage:
cat /proc/key-users
# 1000: 14 14/200 14/200 9kb/20kbThat is UID 1000, 14 keys currently held, 9KB of the 20KB limit. Services that create keys without revoking them hit the quota. Always revoke when done, or rely on TTL expiry. The defaults are configurable via those sysctl paths if 200 keys is too tight for your use case.
Keys you have permission to view appear in /proc/keys. Keys that exist but which you cannot view do not. Useful when debugging why request_key is failing:
cat /proc/keys
# 1d6c69a5 I--Q--- 1 perm 3f010000 1000 1000 user: myapp:db:passwordFields: key ID, flags (I = instantiated, Q = quota counted), link count, flags again, permissions, UID, GID, type, description.
The request-key mechanism
request_key with a non-NULL callout string will invoke /sbin/request-key in userspace if the key is not found in the hierarchy. This is how the kernel fetches Kerberos tickets and NFS session keys: the kernel asks userspace for the credential, userspace obtains it, calls keyctl instantiate, and the original kernel request completes.
For application use: pass NULL as the callout unless you have specifically configured /etc/request-key.conf. A non-NULL callout on a cache miss forks a process running as your UID and executes /sbin/request-key. That is a surprise you do not want and it costs a fork.
What the keyring does not do
Key payloads for the user type are kmalloc’d kernel memory but are not pinned with the equivalent of mlock. On a memory-pressured system with swap enabled they can theoretically be written out. For secrets that must never touch disk: store the canonical secret in the keyring, read it into a memfd_secret region for active use. The memfd_secret post covers that side of the problem.
The keyring also does not help if the process itself is compromised. Arbitrary code execution in your webserver with access to the calling process’s keyrings means an attacker can call keyctl(KEYCTL_READ, ...) and read your secret. Pair the keyring with seccomp-bpf filters that block add_key, request_key, and keyctl except during the init phase, and with PR_SET_NO_NEW_PRIVS to prevent privilege escalation that would bypass the filter.
Finally: user keys are readable by any code running as the same UID that knows the key serial or description. That is a feature, not a bug, but it means key descriptions should be opaque enough that a generic scraper cannot enumerate them. myapp_db_password is worse than a UUIDv4 with a private mapping table in your application.
Why bother
Versus env vars: payloads are not in /proc/PID/environ, not in core dumps, not in ps output, and you can revoke them without killing the process. A deploy that rotates credentials can invalidate the old key and store the new one; the application picks up the new value on next request_key call.
Versus a secrets file on disk: there is no file to cat, no permission bug that exposes it to other services on the same mount, no disk forensics that recovers it after shred. The secret exists in kernel memory and nowhere else.
Is it a clean API? No. The quota defaults are low. The permission bitmask is non-obvious. user keys are swappable in theory. libkeyutils is something most teams have never heard of, and the TPM-backed types add real operational complexity.
But it is already in the kernel. It costs nothing to use. It fixes the most common credential leaks in one shot. If you are still handing a database password to a subprocess through an environment variable, the fix is apt install libkeyutils-dev and ten lines of C.
Kerrisk covers the keyring API in Chapter 32 of The Linux Programming Interface. The man pages keyrings(7), keyctl(2), add_key(2), and request_key(2) fill in the gaps.