Re: aio tests failing on newer Linux kernels

On Tue, Jul 28, 2026 at 7:15 AM Xuneng Zhou <xunengzhou(at)gmail(dot)com> wrote:
>
> Hi,
>
> On Tue, Jul 28, 2026 at 12:00 PM Alexander Lakhin <exclusion(at)gmail(dot)com> wrote:
> >
> > Hello Aleksander and Michael,
> >
> > 28.07.2026 05:24, Michael Paquier пишет:
> >
> > On Mon, Jul 27, 2026 at 04:36:24PM +0300, Aleksander Alekseev wrote:
> >
> > AI suggests the new behavior is caused by the "io_uring: use region api
> > for SQ/CQ" change landed in the kernel version 6.14.
> >
> > No idea about the AI argument, but..
> >
> > Two cents: I'm on Ubuntu 24.04 with Linux 6.14.0-37-generic and
> > liburing 2.5 and the tests pass fine for me.
> >
> > You are not alone. Various buildfarm members are also turning red
> > with random failures across multiple tests because of this issue.
> >
> >
> > I think, I've found the difference -- on Debian I'm seeing:
> > user(at)debian:~$ ulimit -l
> > 8192
> > vs
> > vagrant(at)ubuntu:~$ ulimit -l
> > 2044616
> >
> > Thus, with "ulimit -l 8192" set on Ubuntu 24.04, I've got the aio failures there as well:
> > Linux ubuntu 6.14.0-37-generic #37~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Nov 20 10:25:38 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
> >
> > $ meson test test_aio_* -j 30
> > ...
> > Ok: 101
> > Expected Fail: 0
> > Fail: 19
> > Unexpected Pass: 0
> >
> > Also reproduced on Fedora, with:
> > Linux localhost.localdomain 6.17.10-100.fc41.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Dec 1 16:10:21 UTC 2025 x86_64 GNU/Linux
> > $ ulimit -l
> > 8192
> > (default)
>
> Out of interest, I let Sol do some investigations for this issue. The
> conclusion it reached seems to align with that of the AI tool used by
> Alexander. However, I lacked the expertise to fully review/confirm it.
> Not sure whether it is acceptable to post a 'could-be-sloppy' report
> here.

Interesting. To me that output makes a lot of sense. We are having plenty of
io_urings allocated (one for each connection up to max_connections), they are
small, but the tests allocate many separate clusters too and all of this hits
the ulimit of max locked memory. I think the 8MB is the norm (e.g.
systemd-system.conf(5) says "DefaultLimitMEMLOCK= defaults to 8MB and there
can be per-service LimitMEMLOCK too), but Ubuntu seems to be way less
conservative and more open as it's mostly for users with X11/audio daemons
(pipewire) and stuff like that.

Sadly soft:hard limit is the same, so we won't be able to bump it like in [1]

Dunno what to do, it's somewhat silent(?) kernel change: reduce parallelism of
the tests maybe with even lower max_connections, bump DefaultLimitMEMLOCK ordo
it in per-service (but dunno how's buildfarm env looks like), bring back older
kernels (but what about security fixes?)

If that AI report is to believed we are going to need to update docs as it
will be hit in the wild too:

- as max_connections=1k + io_uring won't possible on newest kernels (but I
haven't verified the math behind it)

- on non-standard > 4kB page systems (ARM?) same thing might be happening
(2 pages won't 8192 there, but much more hitting earlier the ENOMEM

-J.

[1] - https://www.postgresql.org/message-id/DJS6KELUH3T0.2DGCBB9I190X%40jeltef.nl