Re: aio tests failing on newer Linux kernels

On Tue, Jul 28, 2026 at 3:04 PM Jakub Wartak
<jakub(dot)wartak(at)enterprisedb(dot)com> wrote:
>
> On Tue, Jul 28, 2026 at 7:15 AM Xuneng Zhou <xunengzhou(at)gmail(dot)com> wrote:
> >
> > Hi,
> >
> > On Tue, Jul 28, 2026 at 12:00 PM Alexander Lakhin <exclusion(at)gmail(dot)com> wrote:
> > >
> > > Hello Aleksander and Michael,
> > >
> > > 28.07.2026 05:24, Michael Paquier пишет:
> > >
> > > On Mon, Jul 27, 2026 at 04:36:24PM +0300, Aleksander Alekseev wrote:
> > >
> > > AI suggests the new behavior is caused by the "io_uring: use region api
> > > for SQ/CQ" change landed in the kernel version 6.14.
> > >
> > > No idea about the AI argument, but..
> > >
> > > Two cents: I'm on Ubuntu 24.04 with Linux 6.14.0-37-generic and
> > > liburing 2.5 and the tests pass fine for me.
> > >
> > > You are not alone. Various buildfarm members are also turning red
> > > with random failures across multiple tests because of this issue.
> > >
> > >
> > > I think, I've found the difference -- on Debian I'm seeing:
> > > user(at)debian:~$ ulimit -l
> > > 8192
> > > vs
> > > vagrant(at)ubuntu:~$ ulimit -l
> > > 2044616
> > >
> > > Thus, with "ulimit -l 8192" set on Ubuntu 24.04, I've got the aio failures there as well:
> > > Linux ubuntu 6.14.0-37-generic #37~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Nov 20 10:25:38 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
> > >
> > > $ meson test test_aio_* -j 30
> > > ...
> > > Ok: 101
> > > Expected Fail: 0
> > > Fail: 19
> > > Unexpected Pass: 0
> > >
> > > Also reproduced on Fedora, with:
> > > Linux localhost.localdomain 6.17.10-100.fc41.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Dec 1 16:10:21 UTC 2025 x86_64 GNU/Linux
> > > $ ulimit -l
> > > 8192
> > > (default)
> >
> > Out of interest, I let Sol do some investigations for this issue. The
> > conclusion it reached seems to align with that of the AI tool used by
> > Alexander. However, I lacked the expertise to fully review/confirm it.
> > Not sure whether it is acceptable to post a 'could-be-sloppy' report
> > here.
>
>
> Interesting. To me that output makes a lot of sense. We are having plenty of
> io_urings allocated (one for each connection up to max_connections), they are
> small, but the tests allocate many separate clusters too and all of this hits
> the ulimit of max locked memory. I think the 8MB is the norm (e.g.
> systemd-system.conf(5) says "DefaultLimitMEMLOCK= defaults to 8MB and there
> can be per-service LimitMEMLOCK too), but Ubuntu seems to be way less
> conservative and more open as it's mostly for users with X11/audio daemons
> (pipewire) and stuff like that.
>
> Sadly soft:hard limit is the same, so we won't be able to bump it like in [1]
>
> Dunno what to do, it's somewhat silent(?) kernel change: reduce parallelism of
> the tests maybe with even lower max_connections, bump DefaultLimitMEMLOCK ordo
> it in per-service (but dunno how's buildfarm env looks like), bring back older
> kernels (but what about security fixes?)
>
> If that AI report is to believed we are going to need to update docs as it
> will be hit in the wild too:
>
> - as max_connections=1k + io_uring won't possible on newest kernels (but I
> haven't verified the math behind it)
>
> - on non-standard > 4kB page systems (ARM?) same thing might be happening
> (2 pages won't 8192 there, but much more hitting earlier the ENOMEM
>
> -J.
>
> [1] - https://www.postgresql.org/message-id/DJS6KELUH3T0.2DGCBB9I190X%40jeltef.nl

I've started to look into this issue seriously. The upstream has
landed a fix on its side citing this thread[1] [2]. Let's see what we
can do on our side.

[1] https://github.com/torvalds/linux/commit/b59cabf9b4acb65f3f9968087ca0e18010d85076
[2] https://lore.kernel.org/io-uring/20261006125732(dot)3425762-1-hengyul(at)cs(dot)unc(dot)edu/T/#u

--
Regards,
Xuneng Zhou
HighGo Software Co., Ltd.