Re: aio tests failing on newer Linux kernels

On Sun, Nov 8, 2026 at 5:03 AM Xuneng Zhou <xunengzhou(at)gmail(dot)com> wrote:
>
> On Tue, Jul 28, 2026 at 3:04 PM Jakub Wartak
> <jakub(dot)wartak(at)enterprisedb(dot)com> wrote:
> >
> > On Tue, Jul 28, 2026 at 7:15 AM Xuneng Zhou <xunengzhou(at)gmail(dot)com> wrote:
> > >
> > > Hi,
> > >
> > > On Tue, Jul 28, 2026 at 12:00 PM Alexander Lakhin <exclusion(at)gmail(dot)com> wrote:
> > > >
> > > > Hello Aleksander and Michael,
> > > >
> > > > 28.07.2026 05:24, Michael Paquier пишет:
> > > >
> > > > On Mon, Jul 27, 2026 at 04:36:24PM +0300, Aleksander Alekseev wrote:
> > > >
> > > > AI suggests the new behavior is caused by the "io_uring: use region api
> > > > for SQ/CQ" change landed in the kernel version 6.14.
> > > >
> > > > No idea about the AI argument, but..
> > > >
> > > > Two cents: I'm on Ubuntu 24.04 with Linux 6.14.0-37-generic and
> > > > liburing 2.5 and the tests pass fine for me.
> > > >
> > > > You are not alone. Various buildfarm members are also turning red
> > > > with random failures across multiple tests because of this issue.
> > > >
> > > >
> > > > I think, I've found the difference -- on Debian I'm seeing:
> > > > user(at)debian:~$ ulimit -l
> > > > 8192
> > > > vs
> > > > vagrant(at)ubuntu:~$ ulimit -l
> > > > 2044616
> > > >
> > > > Thus, with "ulimit -l 8192" set on Ubuntu 24.04, I've got the aio failures there as well:
> > > > Linux ubuntu 6.14.0-37-generic #37~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Nov 20 10:25:38 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
> > > >
> > > > $ meson test test_aio_* -j 30
> > > > ...
> > > > Ok: 101
> > > > Expected Fail: 0
> > > > Fail: 19
> > > > Unexpected Pass: 0
> > > >
> > > > Also reproduced on Fedora, with:
> > > > Linux localhost.localdomain 6.17.10-100.fc41.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Dec 1 16:10:21 UTC 2025 x86_64 GNU/Linux
> > > > $ ulimit -l
> > > > 8192
> > > > (default)
> > >
> > > Out of interest, I let Sol do some investigations for this issue. The
> > > conclusion it reached seems to align with that of the AI tool used by
> > > Alexander. However, I lacked the expertise to fully review/confirm it.
> > > Not sure whether it is acceptable to post a 'could-be-sloppy' report
> > > here.
> >
> >
> > Interesting. To me that output makes a lot of sense. We are having plenty of
> > io_urings allocated (one for each connection up to max_connections), they are
> > small, but the tests allocate many separate clusters too and all of this hits
> > the ulimit of max locked memory. I think the 8MB is the norm (e.g.
> > systemd-system.conf(5) says "DefaultLimitMEMLOCK= defaults to 8MB and there
> > can be per-service LimitMEMLOCK too), but Ubuntu seems to be way less
> > conservative and more open as it's mostly for users with X11/audio daemons
> > (pipewire) and stuff like that.
> >
> > Sadly soft:hard limit is the same, so we won't be able to bump it like in [1]
> >
> > Dunno what to do, it's somewhat silent(?) kernel change: reduce parallelism of
> > the tests maybe with even lower max_connections, bump DefaultLimitMEMLOCK ordo
> > it in per-service (but dunno how's buildfarm env looks like), bring back older
> > kernels (but what about security fixes?)
> >
> > If that AI report is to believed we are going to need to update docs as it
> > will be hit in the wild too:
> >
> > - as max_connections=1k + io_uring won't possible on newest kernels (but I
> > haven't verified the math behind it)
> >
> > - on non-standard > 4kB page systems (ARM?) same thing might be happening
> > (2 pages won't 8192 there, but much more hitting earlier the ENOMEM
> >
> > -J.
> >
> > [1] - https://www.postgresql.org/message-id/DJS6KELUH3T0.2DGCBB9I190X%40jeltef.nl
>
> I've started to look into this issue seriously. The upstream has
> landed a fix on its side citing this thread[1] [2]. Let's see what we
> can do on our side.
>
> [1] https://github.com/torvalds/linux/commit/b59cabf9b4acb65f3f9968087ca0e18010d85076
> [2] https://lore.kernel.org/io-uring/20261006125732(dot)3425762-1-hengyul(at)cs(dot)unc(dot)edu/T/#u

Hi Xuneng,

Great find and link! Maybe I am missing something, but I doubt we need to
anything here with because kernel folks fixed it already and we just need
to say to (potential) users to update kernel?

a. the only affected LTS kernel seems to be 6.18.x [1]
b. as per [2] changelog that was fixed in 6.18.58 commit
a6d84b420a84843d837f729c298d98d8b070d314 ("io_uring/kbuf: don't charge
provided buffer rings to RLIMIT_MEMLOCK") from 1 month ago

I've just tried it on a bit older 7.0.x kernel which is buggy (and never got the
fix in the first place):

$ gcc t.c -o t -luring
$ cat t.c # their repro
#include <liburing.h>
#include <stdio.h>
int main(void)
{
static struct io_uring ring[64];
int i;

for (i = 0; i < 64; i++)
if (io_uring_queue_init(4096, &ring[i], 0) < 0)
break;
printf("%d rings\n", i);
return 0;
}

$ ulimit -l # max locked memory,
4036812
$ ./t
64 rings
$ ulimit -l 8192
$ ./t
19 rings
ulimit -l 4036812
bash: ulimit: max locked memory: cannot modify limit: Operation not permitte

.. so yeah I just should bump ulimit or just switch to fixed kernel. Bumping
up ulimit -l - once that's locked down - might require some reconfiguration
of the systemd/OS, but I bet that is the price of running out-dated and buggy
software (or just potentally distros lagging with backports to their kernel
versions)

-J.

[1] - https://kernel.org/category/releases.html
[2] - https://cdn.kernel.org/pub/linux/kernel/v6.x/ChangeLog-6.18.58