Re: A bug in recovery prefetcher, triggering asserts / FATAL: checkpoint request failed

> On Oct 26, 2026, at 07:37, Tomas Vondra <tomas(at)vondra(dot)me> wrote:
>
> Hi,
>
> While hacking on an improvement for the recovery prefetcher, I ran into
> a pre-existing bug present since PG 15. The brief explanation is that
> XLogPrefetcherBeginRead resets a bunch of stuff, but not the pointer
> prefetcher->record. In some circumstances (full I/O queue, ...) that
> happens to confuse XLogPrefetcherReadRecord, triggering failures:
>
> - assert failures on development builds
>
> Assert(lrq_completed(prefetcher->streaming_read) == 0);
>
> - a bunch of weird/uncommon errors on regular builds
>
> LOG: request to flush past end of generated WAL; request 0/030BBD90,
> current position 0/030B9D48
> ERROR: xlog flush request 0/030BBD90 is not satisfied --- flushed
> only to 0/030B9D48
> FATAL: checkpoint request failed
>
> AFAICS it can happen only with reduced maintenance_io_concurrency, but
> I'm not sure about that. Maybe there's a way to generate WAL records
> with enough block images.
>
> The attached patch fixes this by simply resetting the record to NULL in
> XLogPrefetcherBeginRead. It also introduces a new TAP test triggering
> this, using a SP-GiST index built etc.
>
> I'm not sure if we want to add the TAP test, I'm attaching it more as a
> way to reproduce the issue. Without the fix in XLogPrefetcherBeginRead,
> it crashes with errors similar to the attached logs.
>
> regards
>
> --
> Tomas Vondra
> <0001-Forget-the-prefetcher-s-current-record-in-XLogPrefet.patch><failure-2.log><failure-1.log>

I confirmed that the new test from v1 failed without the fix, and passed after patching v1.

So, the bug is that the prefetcher continued examining the stale prefetcher->record and filled the I/O queue without decoding the record at the new read position. The resulting assertion failure terminated the startup process.

The fix is straightforward and looks good to me. Although the failure may be an edge case, clearing the stale pointer is certainly the right thing to do.

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/