Re: BUG #19796: io_uring crashes with specific kernel/liburing versions + io_max_concurrency due to rings overlap

вс, 25 окт. 2026 г. в 18:34, PG Bug reporting form <noreply(at)postgresql(dot)org>:

> The following bug has been logged on the website:
>
> Bug reference: 19796
> Logged by: Alexander Lakhin
> Email address: exclusion(at)gmail(dot)com
> PostgreSQL version: 19rc1
> Operating system: Ubuntu 24.04
> Description:
>
> The following script:
> psql -c "CREATE TABLE t AS SELECT repeat('x', 100) FROM generate_series(1,
> 500000)"
>
> echo "SELECT count(*) FROM t" > /tmp/q.sql
> pgbench -n -f /tmp/q.sql -c 4 -T 60
>
> with these parameters set:
> io_method = io_uring
> io_max_concurrency = 1024
> shared_buffers = 16MB
> max_connections = 30
>
> crashes on Ubuntu 24.04 as below:
> 2026-10-25 06:03:22.401 UTC [39572] PANIC: io_uring submit submitted only
> 0
> of 1
> 2026-10-25 06:03:22.401 UTC [39572] STATEMENT: SELECT count(*) FROM t
> 2026-10-25 06:03:22.401 UTC [39523] ERROR: io_uring submit submitted only
> 0
> of 1
> 2026-10-25 06:03:22.401 UTC [39523] CONTEXT: parallel worker
>
> or
> TRAP: failed Assert("ioh->state == PGAIO_HS_SUBMITTED"), File: "aio.c",
> Line: 530, PID: 39816
> ExceptionalCondition at assert.c:51:13
> pgaio_io_process_completion at aio.c:532:2
> pgaio_uring_drain_locked at method_io_uring.c:565:20
> pgaio_uring_wait_one at method_io_uring.c:643:4
> pgaio_io_wait at aio.c:623:6
> pgaio_wref_wait at aio.c:999:1
> WaitReadBuffers at bufmgr.c:1831:17
> read_stream_next_buffer at read_stream.c:1191:17
> heap_fetch_next_buffer at heapam.c:743:16
> heapgettup_pagemode at heapam.c:1110:8
> heap_getnextslot at heapam.c:1484:3
> table_scan_getnextslot at tableam.h:1105:1
> SeqNext at nodeSeqscan.c:90:5
> ...
>
> With this logging added:
> --- a/src/backend/storage/aio/method_io_uring.c
> +++ b/src/backend/storage/aio/method_io_uring.c
> @@ -357,6 +357,12 @@ pgaio_uring_shmem_init(void *arg)
> struct io_uring_params p = {0};
>
> ret = io_uring_queue_init_mem(io_max_concurrency,
> &context->io_uring_ring, &p, ring_mem_next, ring_mem_remain);
> +if (ret > 0)
> +{
> + int overlap = (char *) (context->io_uring_ring.sq.array +
> p.sq_entries) - (ring_mem_next + ret);
> + if (overlap > 0)
> + elog(LOG, "!!!pgaio_uring_shmem_init| io_uring ring %d
> overlaps the next ring by %d bytes", contextno, overlap);
> +}
>
> ring_mem_remain -= ret;
> ring_mem_next += ret;
>
> I can see:
> 2026-10-25 06:04:51.347 UTC [39804] LOG: !!!pgaio_uring_shmem_init|
> io_uring ring 0 overlaps the next ring by 64 bytes
> 2026-10-25 06:04:51.347 UTC [39804] LOG: !!!pgaio_uring_shmem_init|
> io_uring ring 1 overlaps the next ring by 64 bytes
> ...
> 2026-10-25 06:04:51.352 UTC [39804] LOG: !!!pgaio_uring_shmem_init|
> io_uring ring 70 overlaps the next ring by 64 bytes
> 2026-10-25 06:04:51.352 UTC [39804] LOG: !!!pgaio_uring_shmem_init|
> io_uring ring 71 overlaps the next ring by 64 bytes
> 2026-10-25 06:04:51.354 UTC [39804] LOG: starting PostgreSQL 20devel on
> x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 13.3.0-6ubuntu2~24.04.1)
> 13.3.0, 64-bit
>
> I observed this on Ubuntu machines with kernels:
> Linux a 7.0.0-1014-aws #14~24.04.1-Ubuntu SMP PREEMPT Fri Sep 18 21:50:22
> UTC 2026 aarch64 aarch64 aarch64 GNU/Linux
> Linux b 6.14.1-061401-generic #202504071048 SMP PREEMPT_DYNAMIC Mon Apr 7
> 11:20:41 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
> and liburing 2.5. The issue is not reproduced after
> https://github.com/axboe/liburing/commit/9f4464d.
>
>
>
>
>
Hi, Alexander!

Thanks for the report.

PostgreSQL stores one io_uring ring per backend in one shared-memory
buffer. The next ring starts at the byte count io_uring_queue_init_mem()
returns. Before liburing 2.7 that count leaves out the io_rings header. The
kernel's ring is longer, and the next ring overlaps its tail.

The real length is in the parameters the kernel fills in. cq_off.cqes is
where the CQE array starts, and that offset already includes the header.
Add the CQE array, align to the cache line, then add the SQ index array if
the kernel created one. On a 1024-entry ring the missing part is 64 bytes.
With liburing 2.5 it is the end of the SQ index array.
Liburing 2.6 has no index array, and those 64 bytes are the end of the CQE
array. Liburing 2.7 includes the header, so the returned count is already
enough.
A fixed extra 64 bytes would also move a correct liburing, and the next
ring would no longer start on a page. The kernel rejects such an address.
The patch keeps whichever is further, the computed end or liburing's count,
and rounds that up to a page. If liburing's count is already past that end,
the reservation stays as it is. Without io_uring_queue_init_mem() and
IORING_SETUP_NO_MMAP this code is not compiled. Each ring still gets its
own mapping.

Checked with io_method=io_uring and io_max_concurrency=1024. liburing 2.5
reported 102400 bytes per ring.
Unpatched, pgbench -c 4 hit both "io_uring submit submitted only 0 of 1"
and Assert(ioh->state == PGAIO_HS_SUBMITTED).
Patched, the server reserved 106496 bytes and the same run finished.
On liburing 2.7 the reservation stayed 102400 bytes and the run finished.

Alexander, please try the builds you still have. liburing 2.5, 2.6 and 2.7,
on both machines from the report.
I only ran 2.5 and 2.7 on x86_64.

Andres in CC.

--
Regards,
Rachitskiy Andrey