pgsql: Re-register LSN waiters after stale wakeups

Re-register LSN waiters after stale wakeups

WaitLSNWakeup() removes a waiter from the heap before setting its latch.
If the position that caused the wakeup moves backwards before the waiter
rechecks it, as can happen when WAL streaming restarts, the waiter may
sleep again while no longer registered. Subsequent WAL progress then
cannot wake it.

When an unmet waiter finds that it is no longer in the heap, add it back
and restart the loop. Rereading the position after registration also
prevents missing an advance between the previous read and the re-add.

Process interrupts before re-registering rather than after, so that a
wakeup which goes stale again cannot postpone cancellation, however often
it repeats. As a side effect, a pending cancel now wins over an expired
timeout, which previously reported a timeout instead.

Add deterministic TAP coverage that simulates a stale standby_write
wakeup without advancing the actual write or replay positions.

Author: Xuneng Zhou <xunengzhou(at)gmail(dot)com>
Reviewed-by: Alexander Korotkov <aekorotkov(at)gmail(dot)com>
Discussion: https://postgr.es/m/CABPTF7UtW_cAa%3DQh4RDfKiUqu3pJJE22ai9tbWJVERbeRyssLw%40mail.gmail.com
Backpatch-through: 19

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/70b7e968d5773026ea5105ae7ff23d362bf9723d

Modified Files
--------------
src/backend/access/transam/xlogwait.c | 37 +++++++-
src/test/modules/Makefile | 1 +
src/test/modules/meson.build | 1 +
src/test/modules/test_wait_lsn/Makefile | 21 +++++
src/test/modules/test_wait_lsn/meson.build | 22 +++++
.../modules/test_wait_lsn/test_wait_lsn--1.0.sql | 14 +++
src/test/modules/test_wait_lsn/test_wait_lsn.c | 99 ++++++++++++++++++++++
.../modules/test_wait_lsn/test_wait_lsn.control | 4 +
src/test/recovery/Makefile | 3 +-
src/test/recovery/t/049_wait_for_lsn.pl | 85 +++++++++++++++++++
10 files changed, 284 insertions(+), 3 deletions(-)