RE: Recovery at replica stuck because recovery incorrectly trusts an old high-water mark

v1 rewound walrcv->flushedUpto in RequestXLogStreaming() whenever the
(segment-rounded) start pointer was behind it.  That was too broad:

  * RequestXLogStreaming() rounds recptr down to a segment boundary
    before the comparison, so the rewind fired on ordinary reconnects,
    not only after a failed read.
  * pg_last_wal_receive_lsn() jumped backward.
  * Cascading walsenders can hit Assert(sentPtr <= SendRqstPtr) when
    GetStandbyFlushRecPtr() drops below a position they already sent.
  * Kyotaro Horiguchi already argued against a blanket flushedUpto
    rewind in 2021 [1].

v2 keeps flushedUpto monotonic.  Startup uses a separate
applyFlushedUpto for "is this streamed WAL readable?".  That pointer is
reset only after lastSourceFailed on archive/pg_wal, when the unrounded
restart LSN is still behind flushedUpto on the same timeline. Healthy
reconnects (including pendingWalRcvRestart after SIGHUP /
primary_conninfo change) keep the old apply pointer, so replay is not
stalled on WAL that is already on disk.  After lastSourceFailed,
WaitForWALToBecomeAvailable() still tries archive/pg_wal first; the
reset only happens if those sources also fail.

The TAP test is now t/058_stream_repair.pl (053 and later numbers were
taken on current master).  It checks both:

  1. primary_conninfo reload with apply lag does not move
     pg_last_wal_receive_lsn() backward.
  2. A corrupt unreplayed record after receive has moved into the next
     segment is replaced via streaming, instead of looping on the local
     copy.

I have not re-run the recovery TAP suite against this tree (no local
install of current master).

[1]
https://postgr.es/m/20210329.105441.1978082841561262877.horikyota.ntt@gmail.com

Konstantin Knizhnik (1):
  Keep flushedUpto monotonic; reset applyFlushedUpto after a failed
    read.

 src/backend/access/transam/xlogrecovery.c  |  38 ++++-
 src/backend/replication/walreceiver.c      |   2 +
 src/backend/replication/walreceiverfuncs.c |  45 +++++-
 src/include/replication/walreceiver.h      |  16 ++-
 src/test/recovery/meson.build              |   1 +
 src/test/recovery/t/058_stream_repair.pl   | 158 +++++++++++++++++++++
 6 files changed, 255 insertions(+), 5 deletions(-)
 create mode 100644 src/test/recovery/t/058_stream_repair.pl