BUG #19396: Standby and DR site replication broken with PANIC: WAL contains references to invalid pages messge

The following bug has been logged on the website:

Bug reference: 19396
Logged by: Ishan Joshi
Email address: ishanjoshi(at)live(dot)com
PostgreSQL version: 16.9
Operating system: ubuntu on Kubernetes
Description:

Hi Team,
I found an issue with PG 16.9 patroni setup where our sytandby node
replication and disaster replication site replication broken with below
error. It looks like WAL corruption which later part of archive file as
well.

CONTEXT: WAL redo at 184F3/F248B6F0 for Heap/LOCK: xmax: 2818115117, off:
35, infobits: [LOCK_ONLY, EXCL_LOCK], flags: 0x00; blkref #0: rel
1663/33195/410203483, blk 25329"
PANIC: WAL contains references to invalid pages"
CONTEXT: WAL redo at 184F3/F248B6F0 for Heap/LOCK: xmax: 2818115117, off:
35, infobits: [LOCK_ONLY, EXCL_LOCK], flags: 0x00; blkref #0: rel
1663/33195/410203483, blk 25329"
WARNING: page 25329 of relation base/33195/410203483 does not exist"
INFO: no action. I am (pg-patroni-node1-0), a secondary, and following a
leader (pg-patroni-node2-0)"
[61]LOG: terminating any other active server processes"
[61]LOG: startup process (PID 72) was terminated by signal 6: Aborted"
[61]LOG: shutting down due to startup process failure"
[61]LOG: database system is shut down"
INFO: establishing a new patroni heartbeat connection to postgres"
INFO: Lock owner: pg-patroni-node2-0; I am pg-patroni-node1-0"
WARNING: Retry got exception: connection problems"
WARNING: Failed to determine PostgreSQL state from the connection, falling
back to cached role"
INFO: Error communicating with PostgreSQL. Will try again later"
WARNING: Postgresql is not running."

Primary db was not impacted, however standby node and DR site replication
broken, I tried to reinit with latest backup + archive loading from
pgbackrest backup but it fails with same error once the corrupt wal/archive
file applying the changes. I had to reinit with pgbasebackup with 40TB
database which took about 45 hrs of time.

Looks like some RACE condition happend to WAL file that generate the issue.
looks like potential bug of it.