Persist slot invalidations before publishing them

Hi hackers,

while reviewing [1], I hit an issue due to the fact that an inactive replication
slot is marked invalid in shared memory before its new state is persisted.

If ReplicationSlotSave() errors before replacing the state file, the slot is
invalid in shared memory but still valid on disk. That sounds problematic as the
resource horizon computations could stop accounting for the slot, remove required
WAL or rows, and then an immediate restart would restore the old valid slot image.

The same issue exists in synchronize_one_slot(): it copies the invalidation from
the remote slot into the local synchronized slot before saving it. In that case,
a save error also prevents a direct retry because the next synchronization sees
the local slot as already invalid and skips it.

The InvalidatePossiblyObsoleteSlot() ordering seems to come from c6550776394e.
4ae08cd5fd19 later made those invalidations persistent but kept the same ordering.

PFA a patch series to $SUBJECT.

It introduces ReplicationSlotPersistInvalidation(), which creates an invalidated
copy of the acquired slot and writes it while the shared slot remains valid.

That means that a concurrent slot saver either writes the old valid state before
the invalidation operation, or waits and snapshots the invalid state after it has
been published. If the invalidated image can not be written, both the shared and
on disk states remain valid.

This is the same kind of idea used of effective_catalog_xmin and in 3741f2a09d52.

The patch series is organized that way:

0001: persist InvalidatePossiblyObsoleteSlot() invalidations before publishing
them.

Add ReplicationSlotPersistInvalidation(), which writes and fsyncs an invalidated
copy while the shared slot remains valid. Hold io_in_progress_lock until the
invalidation is published so checkpoints and other slot savers cannot persist
a stale image after publication.

It also adds an injection point and some tests.

0002: do the same for synchronize_one_slot().

It keeps the synchronized slot restart LSN when copying a WAL invalidation,
preserving the current behavior. It also recomputes the xmin and WAL horizons
once the invalidation is durable and visible.

It also adds some test.

Remarks:

1/ there is no new retry mechanism. A later checkpoint or synchronization
naturally retries because the shared slot remains valid after the error.

2/ the patches change neither the on disk slot format nor the ReplicationSlot
shared memory layout. It has been done that way to ease the back patching.

3/ I think 0001 should be backpatched down to 14. 14 and 15 would probably need
some adaptations though (I did not look in detail yet).

4/ 0002 should be backpatched down to 17, where failover slot synchronization was
introduced.

[1]: https://postgr.es/m/CALj2ACUi0LeqKzomuXNPFsekzuQ%2BXbWZ5RamnO6tDzG4i1-KLw%40mail.gmail.com

Regards,

--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com