Standby stuck after walreceiver restart just before a WAL segment boundary

Hello,

We've found a bug where a streaming standby gets stuck if its
walreceiver restarts
while the last record it has received ends 8, 16 or 24 bytes before
the end of a WAL segment, so that the next record header reaches the
segment boundary. This issue reproduces on current master and the code
is the same on all supported branches.

To read that header, XLogPageRead() asks WaitForWALToBecomeAvailable()
for WAL up to the end of the header, or the end of the page if the
header crosses it. Here that is exactly the end of the segment.
WaitForWALToBecomeAvailable() passes that end position (RecPtr) to
RequestXLogStreaming() as the start point, and RequestXLogStreaming()
rounds it down to a segment start. That's a no-op here, so the new
walreceiver starts streaming at the next segment and never fetches the
end of the current one:

LOG: walreceiver requested start point 0/05000000 on timeline 1 is
ahead of the upstream server's flush position 0/04FFFFF8, waiting
LOG: started streaming WAL from primary at 0/05000000 on timeline 1
LOG: invalid record length at 0/04FFFFF8: expected at least 24, got 0
[... repeated forever]

The attached patch starts streaming at the beginning of the segment
holding RecPtr - 1, which is the segment of the page being read, as
c06380e976 intended when it switched to RecPtr. Nothing changes unless
RecPtr is exactly on a segment boundary. It adds a TAP test,
t/061_walrcv_restart_segment_boundary.pl, which pads WAL so that the
last record ends 8 bytes before a boundary, restarts the walreceiver by
changing primary_conninfo, and checks where streaming restarts. It
fails on master without the fix and passes with it; I also checked gaps
of 16 and 24 bytes by changing the test. src/test/recovery passes with
make and meson.

Related threads:

- Konstantin's applyFlushedUpto patch [1] lets recovery get past WAL
that is flushed but unreadable, and Jeff asked there how a standby
ends up with such WAL. This is one way. With that patch the test
recovers after a re-read; with this one the skip doesn't happen.
These patches are independent of each other.

- Paul's fix for the end LSN of an xlog-switch record [2] is a
primary-side way into the same state.

- Since b614de4876 [3], the walreceiver waits instead of failing when
its start point is at most one segment ahead of the upstream, so
this case now gets stuck without an error. Before that,
START_REPLICATION failed and was retried until the primary had
written past the boundary, and then the same skip happened.

I think it makes sense to backport the fix since it applies to
REL_15_STABLE through REL_19_STABLE. REL_14_STABLE
has the same code in xlog.c, and I can send a version for it.

[1] https://postgr.es/m/4be52164-66f7-43c0-8ceb-7ce55ba67ac0@garret.ru
[2] https://postgr.es/m/CALj2ACVsQuwDYovODF4qFWfR9Ye4PwcqzJoinigOCkPAzjM0hg@mail.gmail.com
[3] https://postgr.es/m/CA+nrD2cTuTkkX5WXVZengTYYZbAO6zV8K+Tri-R0fbLFuoyMBA@mail.gmail.com

Thanks,
David Shorten