Re: Be strict when request to flush past end of WAL in WaitXLogInsertionsToFinish

Hi,

On Thu, Sep 3, 2026 at 6:29 PM Paul Kim <mok03127(at)gmail(dot)com> wrote:
>
> > Just curious, how did the bogus LSN end up in asyncXactLSN in
> > production when you hit the issue?
>
> Honestly, we still don't know, but the value itself is telling.
>
> The production request was exactly segment boundary + 0x28, i.e.
> SizeOfXLogLongPHD past a segment start. That is precisely what the
> current insert position looks like right after a segment switch,
> before any record has been written into the new segment. So it does
> not look like random corruption; it looks like something captured the
> insert position at that moment and handed it to XLogSetAsyncXactLSN().
> That is also how the TAP test's injector reproduces the incident
> byte-for-byte.
>
> I went looking for an in-core path that could do this naturally on the
> affected version and came up empty: fault-free segment switches
> (~100 runs), async commits, LogStandbySnapshot() calls, and top-level
> aborts never produced the "past end of generated WAL" warning in my
> testing. The affected installation does load third-party preload
> libraries; auditing those for WAL-related symbol use is on my list but
> has not been done yet, so I cannot rule the source in or out.
>
> Either way, I think the fix stands on its own: whatever plants the
> value, WaitXLogInsertionsToFinish() already detects and clamps it, and
> XLogBackgroundFlush() discarding that clamp is what turns one bad
> request into the standby's prev-link retry loop.

Thanks for the patches. I don't see a CF entry yet, so I created one:
https://commitfest.postgresql.org/patch/7294/. Feel free to add
yourself as an author.

I went through this thread today. Here's my take.

What exactly caused the wait-for-any-in-progress-insertions-to-finish
to receive an LSN past the end of the generated WAL is one problem.
And, when that happens for whatever reason, the walwriter not honoring
the adjusted position in the caller and blindly writing such WAL to
WAL files is another problem.

In this case, although we don't yet know the root cause for the first
problem, which could be not necessarily the async commit/abort LSN
being wrong but could be anyone else setting up an LSN beyond what's
written in XLogCtl->LogwrtRqst, I think fixing the second problem is
the right direction (as the patch does here). The backend doing WAL
write already honors the adjusted position, so the walwriter missing
it needs to be fixed too. If the backend gets to write the WAL before
the walwriter, it would not have written this WAL record because
wait-for-any-in-progress-insertions-to-finish in XLogFlush() honors
the adjusted position. This matters because the consequences on the
standby are hard to deal with in production, stuck WAL replay, vacuum
issues on the primary, and possibly failovers.

If I understand correctly, you identified that the walwriter is the
problem by looking at the pid from the "request to flush past end of
generated WAL" log message, right? Nice find.

Also, I'm curious, how did the standby get out of the stuck error loop
"record with incorrect prev-link"?

Also, did you observe any "xlog flush request %X/%08X is not satisfied
--- flushed only to" or other messages on the primary? And I believe
if the primary had crashed before checkpointing this WAL record, it
would have also been stuck in a similar error loop, right?

A few comments on the patch.

1/ Nit. How about using "adjusted" instead of "clamped" in the
comments and commit message?

+ * if 'upto' is past the end of reserved WAL, the request is clamped to the
+ /* honor the clamp if the request was past the end of reserved WAL */
WaitXLogInsertionsToFinish() clamps a request that is past the end of

2/ Why do we need to check the adjusted LSN against the requested LSN
again? Also, is there a reason to compare it with the flush LSN? Why
not just assign the adjusted LSNs like XLogFlush() does?

+ /* honor the clamp if the request was past the end of reserved WAL */
+ if (insertpos < WriteRqst.Write)
+ {
+ WriteRqst.Write = insertpos;
+ if (WriteRqst.Flush > insertpos)
+ WriteRqst.Flush = insertpos;
+ }

3/ Do we need similar adjusted handling in AdvanceXLInsertBuffer()? I
don't think so because there the whole old page from the WAL buffer is
written anyway. Just want to clarify.

--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com