Re: pg ignores wal files in pg_wal, and instead tries to load them from archive/primary

On Thu, Sep 29, 2022 at 05:51:02PM +0200, hubert depesz lubaczewski wrote:
> What am I missing? what is wrong? How can I fix it? The problem is not fixing
> *this server*, because we are in process of upgrading LOTS and LOTS of servers,
> and I need to know what is broken/how to work around it.

So, your backup_label file points to 00000001000007E4000000A0 as being
the first segment to begin recovery from. This means that the startup
process is able to correctly kick recovery using the contents of the
local pg_wal/ from 00000001000007E4000000A0 to 00000001000007E800000066.
Once 00000001000007E800000067 cannot be read, the startup process
fails to read this segment and decides to switch the source to get WAL
from as streaming. The order should be local pg_wal/, then archiving
and finally streaming if all are configured, as far as I recall.

What's the error you are seeing when reading 00000001000007E800000067?
Is pg_waldump able to understand its internals? This could point out
to an issue with xlogreader.c, for example.

Another thing that has changed in this area is related to continuation
records. What are the WAL contents at the end of
00000001000007E800000066? Do you spot an OVERWRITE_CONTRECORD when
using pg_waldump on it?
--
Michael