# io_uring `ENOMEM` on Linux 6.14+: `RLIMIT_MEMLOCK` accounting and PostgreSQL AIO failures

**Date:** 2026-07-28  
**Revision:** 3 — incorporates review corrections and Alexander Lakhin's Ubuntu/Fedora reproductions  
**Thread:** [aio tests failing on newer Linux kernels](https://www.postgresql.org/message-id/flat/667f7dcf-404b-4898-a426-57ca6e0617f6@gmail.com)

Evidence labels used below:

- **[S]** confirmed from Linux or PostgreSQL source
- **[O]** directly observed in logs or experiments
- **[I]** inferred explanation consistent with the confirmed mechanism and observations

## Executive summary

**Mechanism confirmed; trigger reproduced; buildfarm attribution remains one host-specific check short of direct proof.**

Linux 6.14 routed io_uring SQ/CQ storage through the region API. As a result, both kernel-allocated and user-provided ring pages now pass through `__io_account_mem()`, which charges the UID-wide `user->locked_vm` counter and rejects an allocation with `-ENOMEM` when it would exceed the calling process's soft `RLIMIT_MEMLOCK` **[S]**. Linux 5.12 had deliberately stopped charging ring arrays to this limit and used memcg accounting instead **[S]**.

PostgreSQL amplifies the change because the postmaster eagerly creates one io_uring instance for every potential backend and relevant auxiliary-process slot, rather than only for running processes **[S]**. Under repository-default non-streaming TAP settings, a typical AIO test node creates 132 rings, which consume about 1.03 MiB of charged memory on a 4 KiB-page system. Concurrent clusters owned by one UID draw from the same `user->locked_vm` counter.

Alexander Lakhin has now provided direct limit-manipulation evidence **[O]**:

- Debian: `ulimit -l` reports **8192 KiB**.
- Ubuntu 24.04: `ulimit -l` reports **2044616 KiB**.
- On Ubuntu 24.04 with Linux `6.14.0-37-generic`, lowering the limit to 8192 KiB and running `meson test test_aio_* -j 30` produced **19 failures out of 120 tests**.
- Fedora 41 with Linux `6.17.10-100.fc41.x86_64` also reproduces the failures with its observed default of 8192 KiB.

This substantially strengthens the causal diagnosis: the kernel version determines whether ring memory is charged, while the inherited `RLIMIT_MEMLOCK` determines whether the workload crosses the failure threshold. The specific buildfarm-host attribution is still technically an inference until that host's effective limits are measured and the same workload is rerun after raising them.

## 1. Kernel mechanism [S]

The accounting policy has changed twice:

| Kernel versions | SQ/CQ ring-memory accounting |
|---|---|
| Linux 5.11 and earlier | charged against `RLIMIT_MEMLOCK` |
| Linux 5.12–6.13 | memcg only (`GFP_KERNEL_ACCOUNT`) |
| Linux 6.14 and later | charged against `RLIMIT_MEMLOCK` again |

Linux 5.12 commit [`26bfa89e25f4`](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=26bfa89e25f4), *io_uring: place ring SQ/CQ arrays under memcg memory limits*, deliberately removed rlimit accounting from ring arrays. Its stated policy was to use memcg for the rings while retaining memlock accounting for registered buffers.

The Linux 6.14 behavior results from the region-API conversion:

- [`dfbbfbf19187`](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=dfbbfbf19187) introduced `io_create_region()` in Linux 6.13, initially for user-provided regions.
- [`8078486e1d53`](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8078486e1d53) and [`81a4058e0cd0`](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=81a4058e0cd0) converted SQ and CQ storage to the region API in Linux 6.14.
- [`fc5f22a64649`](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fc5f22a64649), *io_uring/memmap: account memory before pinning*, moved accounting above the branch between user-provided and kernel-allocated memory.

In Linux 6.14 and later, every ring region therefore executes the equivalent of:

```c
nr_pages = reg->size >> PAGE_SHIFT;
if (ctx->user)
{
        ret = __io_account_mem(ctx->user, nr_pages);
        if (ret)
                return ret;              /* -ENOMEM */
}
```

`__io_account_mem()` compares:

```text
user->locked_vm + requested_pages
```

against:

```text
calling_process.soft_RLIMIT_MEMLOCK
```

and atomically increments the UID-wide counter if the check succeeds. `io_free_region()` releases the charge when the ring is destroyed, so only concurrently live charged regions contribute.

### The precise accounting model

The model is a hybrid:

- **Usage is shared per UID:** `user->locked_vm` belongs to `struct user_struct`.
- **The ceiling is process-local:** each allocation is checked against the calling process's current soft `RLIMIT_MEMLOCK`.

Consequently, two processes owned by the same UID can have different limits while competing against the same accumulated usage counter. Saying simply that the limit is either “per process” or “per user” misses this distinction.

### Capability bypass

`ctx->user` is populated only when the caller lacks `CAP_IPC_LOCK` in the **initial user namespace**:

```c
if (!ns_capable_noaudit(&init_user_ns, CAP_IPC_LOCK))
        ctx->user = get_uid(current_user());
```

A caller holding that capability is not charged. “Root bypasses the limit” is therefore too broad: container root or root in a noninitial user namespace may still be charged, while a non-root caller granted the relevant capability may not be.

### Interpretation

The restored accounting appears to be an unintended consequence of the region conversion **[I]**. The Linux 6.14 commit messages do not discuss reversing the Linux 5.12 policy or restoring `RLIMIT_MEMLOCK` accounting for ring arrays. It is justified to call this a behavioral regression relative to Linux 5.12–6.13; whether it was accidental remains an inference for upstream maintainers to adjudicate.

No existing fix or revert was found in the sources searched as of 2026-07-28, and the behavior was still present in the later kernel trees examined. That search result should not be presented as proof that no report exists anywhere.

## 2. Effective limits and direct reproductions [O]

Linux raised its kernel `MLOCK_LIMIT` baseline to 8 MiB in Linux 5.16 through [`9dcc38e2813e`](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9dcc38e2813e). systemd also adopted an 8 MiB default baseline. These defaults are useful context, but they do **not** reliably predict the limit inherited by a particular shell, service, container, CI worker, or buildfarm client.

Alexander's measurements demonstrate that the effective value can differ dramatically even across apparently similar distributions:

| Environment | Kernel | Observed `ulimit -l` |
|---|---|---:|
| Debian test environment | Linux 6.19.6 in the earlier reproduction | 8192 KiB |
| Ubuntu 24.04 | `6.14.0-37-generic` | 2044616 KiB |
| Fedora 41 | `6.17.10-100.fc41.x86_64` | 8192 KiB by default |

On the Ubuntu 24.04 system, lowering the soft limit and rerunning the tests produced the failure:

```text
$ ulimit -l 8192
$ meson test test_aio_* -j 30
...
Ok:                 101
Expected Fail:        0
Fail:                19
Unexpected Pass:      0
```

This is the strongest experimental evidence in the thread so far because it changes the suspected resource limit on the same operating-system/kernel family and makes the failure appear.

For a strict bidirectional A/B demonstration, the exact same `-j 30` command should also be recorded after restoring the original 2044616 KiB limit. The quoted result establishes the low-limit failure, but it does not state the outcome of that identical high-limit run.

When diagnosing a host, record both soft and hard limits:

```text
prlimit --memlock --nofile -p $$
```

An unprivileged shell can lower its soft limit, but it cannot raise it beyond the inherited hard limit.

## 3. Per-ring charge [S/O]

At the ring geometry PostgreSQL uses in the affected tests, an io_uring instance consumes two base pages:

- one page for the combined SQ/CQ ring structures;
- one page for the SQE array.

That is **8 KiB on systems with 4 KiB pages**. Systems with 16 KiB or 64 KiB base pages incur a correspondingly larger minimum charge.

The PostgreSQL logs confirm the 4 KiB-page result directly:

```text
DEBUG: can use combined memory mapping for io_uring, each ring needs 8192 bytes
```

The same 8192-byte value on x86-64 and i386 establishes the same footprint on those observed 4 KiB-page systems; it does not establish architecture independence in general.

`io_max_concurrency` is not permanently fixed at 64. PostgreSQL auto-tunes it and caps it at 64; the configurations discussed here reach that cap.

## 4. PostgreSQL amplification [S]

`pgaio_uring_shmem_init()` runs in the postmaster and creates one ring per potential process slot:

```text
rings = pgaio_uring_procs()
      = MaxBackends + NUM_AUXILIARY_PROCS - MAX_IO_WORKERS
      = MaxBackends + 6

MaxBackends = max_connections
            + autovacuum_worker_slots
            + max_worker_processes
            + max_wal_senders
            + 2 special-worker slots
```

Therefore:

```text
rings = max_connections
      + autovacuum_worker_slots
      + max_worker_processes
      + max_wal_senders
      + 8
```

The important property is that PostgreSQL allocates rings for **potential slots**, not merely for processes that are currently running.

Using the current defaults of 16 autovacuum worker slots and 8 general worker slots:

| Configuration | `max_connections` | `max_wal_senders` | Rings | Charged at 8 KiB/ring |
|---|---:|---:|---:|---:|
| Production defaults | 100 | 10 | **142** | 1,136 KiB (1.109 MiB) |
| Non-streaming TAP node | 100 | 0 | **132** | 1,056 KiB (1.031 MiB) |
| `003_initdb` during `initdb` itself | 100 | 10 | **142** | 1,136 KiB (transient) |
| `004_read_stream` | 8 | 0 | **40** | 320 KiB (0.313 MiB) |
| Production with `max_connections=1000` | 1000 | 10 | **1,042** | 8,336 KiB (8.141 MiB) |

`PostgreSQL::Test::Cluster` explicitly sets `max_wal_senders=0` for non-streaming test nodes. That is why `001_aio` and the post-init `003_initdb` server use 132 rings, not 142; `003_initdb` can still use 142 transiently during its `initdb -c io_method=io_uring` phase.

Both REL_18 and master/REL_19 contain the combined user-provided-memory path based on `io_uring_queue_init_mem()` and `IORING_SETUP_NO_MMAP`. Linux 6.14 charges that path and the ordinary kernel-allocated path identically. The observed branch difference is therefore most plausibly explained by test coverage and concurrency, rather than by one branch escaping the accounting **[I]**.

Each ring also consumes one file descriptor in the postmaster. PostgreSQL already reports `EMFILE` separately, and its source notes that high process counts may require a larger `RLIMIT_NOFILE`.

## 5. Observed PostgreSQL failures [O]

### Standalone and distribution reproductions

- Debian/Linux 6.19.6: 30 parallel copies of `test_aio` produced 27 failures out of 120 tests, all reporting `could not setup io_uring queue: Cannot allocate memory`; `-j 10` passed.
- The same workload did not fail on Linux 6.12.48.
- Ubuntu 24.04/Linux `6.14.0-37-generic`: the environment had a much larger observed memlock limit, but lowering it to 8192 KiB made `meson test test_aio_* -j 30` fail 19 of 120 tests.
- Fedora 41/Linux `6.17.10-100.fc41.x86_64`: reproduced with an observed default limit of 8192 KiB.

These observations jointly establish both relevant axes:

1. the kernel-version boundary is consistent with the source change; and
2. on affected kernels, lowering `RLIMIT_MEMLOCK` exposes the failure.

### Buildfarm observations

The reported `ENOMEM` failures occurred on several buildfarm animals owned by Andres Freund and apparently sharing one host and UID. Multiple animals can overlap even though branches within one animal run sequentially. Four animals reportedly failed `test_aioCheck` within 93 seconds on the same commit.

The buildfarm logs show the same postmaster failure while starting an io_uring node. A separate `EPERM` failure on another animal must not be grouped with these `ENOMEM` failures; it reflects io_uring being disabled or denied rather than memory-accounting exhaustion.

The buildfarm host's effective soft and hard memlock limits are still not recorded in the available logs. Thus:

- the kernel/PostgreSQL mechanism is confirmed **[S]**;
- the memlock trigger is directly reproduced outside the farm **[O]**;
- attribution of every buildfarm failure to the same trigger remains high-confidence but host-specific inference **[I]**.

## 6. Quantitative fit [I]

At 8 KiB per ring, an 8 MiB limit permits at most 1,024 concurrently charged rings, ignoring any charge already consumed by other io_uring users under the same UID.

- Seven 132-ring TAP clusters: 924 rings, about 7.22 MiB — still below the limit.
- Eight 132-ring TAP clusters: 1,056 rings, about 8.25 MiB — above the limit.
- One fully overlapping `001_aio` + `003_initdb` + `004_read_stream` set: approximately 304–314 rings, or 2.38–2.45 MiB.
- Four such overlapping sets: approximately 9.5–9.8 MiB — sufficient by themselves to exceed an 8 MiB limit.
- Three such sets: approximately 7.1–7.4 MiB — close enough that another live ring set or pre-existing same-UID charge can cross the threshold.

This arithmetic explains why failures are concurrency-sensitive and intermittent. The exact number of simultaneously live test clusters was not measured, so estimates of peak demand in the `-j 30` run remain heuristic rather than direct observation.

Alternative explanations fit the evidence less well:

- genuine system-wide memory exhaustion would not naturally match the Linux 6.14 policy boundary, because Linux 6.12/6.13 allocated the same ring pages without this rlimit charge;
- io_uring permission failures return `EPERM`, not `ENOMEM`;
- file-descriptor exhaustion returns `EMFILE`, for which PostgreSQL already emits a distinct hint.

## 7. Remaining confirmation steps

Before presenting the buildfarm attribution as fully proven, perform the following on an affected host:

1. Record the actual soft and hard limits:

   ```text
   prlimit --memlock --nofile -p $$
   ```

2. Run the same high-concurrency workload under the current limit and record the exact `ENOMEM` count.
3. Raise both the soft and hard memlock limits for the service account, without changing the workload, and rerun it.
4. Confirm that the failures disappear or that the failure threshold moves predictably with the limit.
5. Optionally trace `io_uring_setup()` or `__io_account_mem()`:

   ```text
   strace -f -e trace=io_uring_setup ...
   ```

A minimal unprivileged reproducer that creates rings until `ENOMEM`, run unchanged on Linux 6.13 and 6.14 under the same limits, would be ideal for an upstream kernel report.

## 8. Production impact, with qualifications

The failure requires all of the following:

- PostgreSQL was built with liburing support;
- `io_method=io_uring` was selected—the default remains `worker`;
- the running kernel contains the Linux 6.14 accounting behavior or a backport;
- the caller lacks `CAP_IPC_LOCK` in the initial user namespace;
- the inherited memlock limit is low enough relative to the UID-wide accumulated charge;
- the ring footprint is evaluated using the host's actual base page size and configured ring depth.

Two practical scenarios matter.

### One large cluster

With otherwise default slot counts, `max_connections=1000` produces 1,042 rings, which charge about 8.14 MiB on a 4 KiB-page system. That exceeds an 8 MiB memlock limit.

However, each ring is also an fd. With a soft `RLIMIT_NOFILE` around 1,024, the postmaster may encounter `EMFILE` first. Once an administrator raises the fd limit—a routine requirement for a server configured for approximately 1,000 connections—memlock becomes the next startup constraint on Linux 6.14 and later.

### Several moderate clusters under one UID

This is the cleaner operational risk. Each cluster may remain well within its own process-local limits, while all clusters owned by `postgres`, a CI account, or a buildfarm account accumulate ring charges in the same UID-wide counter. Other io_uring-using applications under that UID consume the same pool.

The Ubuntu observation also shows why this is not universal: a system inheriting a roughly 2 GiB memlock limit will not reproduce the 8 MiB threshold. Effective service limits, not distribution names alone, determine exposure.

## 9. Recommendations

### Immediate operational mitigation

Measure the effective soft and hard limits, then raise memlock for affected service or test accounts through the service manager or PAM limits configuration. A finite value with margin, such as 64 MiB for the reported buildfarm workload, is preferable to assuming that `unlimited` is necessary.

Do not treat granting `CAP_IPC_LOCK` broadly to the PostgreSQL service as the primary workaround; raising the resource limit is narrower and easier to reason about.

### PostgreSQL error reporting

Add an `ENOMEM` branch alongside the existing `EPERM`, `EMFILE`, and `ENOSYS` handling, using `ERRCODE_INSUFFICIENT_RESOURCES`. The hint must be conditional rather than definitive because `ENOMEM` can have other causes. Suggested wording:

> io_uring ring memory may count against `RLIMIT_MEMLOCK`, with charged usage shared by processes of the same user. Consider increasing the service's memlock limit.

PostgreSQL can estimate its own ring requirement from `pgaio_uring_procs()` and either the probed per-ring size or a page-size-aware fallback, but it cannot know how much of the UID-wide counter other processes have already consumed.

### Documentation

Document:

- the Linux-version-dependent accounting behavior;
- the UID-wide usage counter versus process-local limit distinction;
- the capability bypass condition;
- the slot-count-driven number of eager rings;
- interaction with `RLIMIT_NOFILE`;
- the fact that vendor kernels and service managers may alter the effective behavior or limits.

### TAP tests

Lower `max_connections` for `001_aio` and `003_initdb`, which do not need 100 connection slots. This reduces test flakiness and resource consumption, but it should accompany—not replace—the operator-facing hint and documentation, because the underlying startup constraint is real.

### Kernel upstream report

Report the behavior neutrally to the io_uring maintainers with:

- a minimal unprivileged reproducer;
- identical Linux 6.13 and 6.14 runs;
- measured soft and hard memlock limits;
- the ring count at failure;
- Linux 5.12 commit `26bfa89e25f4`, which records the prior deliberate policy;
- the Linux 6.14 region-API commits that restored the charge.

The appropriate upstream question is whether restoring rlimit accounting for ring arrays was intended—not an assumption that a revert or exemption is necessarily the correct fix.

### Longer-term PostgreSQL design

The underlying amplifier is one eagerly allocated ring per potential process slot. Longer-term options include lazy ring creation, a bounded shared pool, selective allocation only for process types that can issue AIO, or carefully designed ring sharing. These require performance and synchronization analysis, but they address the structural source of the resource footprint.

## 10. Sources

- PostgreSQL thread: [flat view](https://www.postgresql.org/message-id/flat/667f7dcf-404b-4898-a426-57ca6e0617f6@gmail.com)
- AIO review discussion of `RLIMIT_MEMLOCK`: [20250312035743.f5.nmisch@google.com](https://www.postgresql.org/message-id/20250312035743.f5.nmisch%40google.com)
- Linux commits: [`26bfa89e25f4`](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=26bfa89e25f4), [`9dcc38e2813e`](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=9dcc38e2813e), [`dfbbfbf19187`](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=dfbbfbf19187), [`8078486e1d53`](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8078486e1d53), [`81a4058e0cd0`](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=81a4058e0cd0), [`fc5f22a64649`](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=fc5f22a64649)
- Linux source examined: `io_uring/{io_uring,memmap,rsrc}.c` at v6.12.48, v6.13.12, v6.14, and v6.19.6
- PostgreSQL source examined: `src/backend/storage/aio/method_io_uring.c`, `aio_init.c`, `src/test/perl/PostgreSQL/Test/Cluster.pm`, and `src/test/modules/test_aio/t/*`
- Representative buildfarm material: [kestrel](https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=kestrel&dt=2026-07-25%2010%3A13%3A20), [melonworm](https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=melonworm&dt=2026-07-25%2017%3A18%3A24), [tamandua](https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=tamandua&dt=2026-07-25%2021%3A15%3A22)

**Publication caveat:** the multi-animal timing/ownership table and branch pass counts should be rechecked against stable per-run records before publication. The buildfarm host's effective memlock limits should also be measured and added.