pg_rewind: Remove recovery at the start of rewind

Hi,

I have been exploring whether pg_rewind can avoid running
crash recovery on a target that was stopped without a clean
shutdown, as at the end of rewind changed rel blocks and non
rel files are copied or removed wrt source.

Currently, pg_rewind starts the target in single-user mode to
complete crash recovery before rewind begins. This reads
all WALs from the last checkpoint and does the replay.

The attached POC instead scans target WAL directly.
pg_rewind already scans target WAL from the last common
checkpoint to construct the page map of blocks changed on
the target. This patch extends that processing to locate the
end of target WAL as well, so that pg_rewind can proceed
without starting postgres on the target.

The target is still required to be stopped. The patch rejects
a live target, removes a stale postmaster.pid when appropriate,
and removes --no-ensure-shutdown because pg_rewind no longer
runs crash recovery.

For locating the checkpoint preceding the divergence point, the normal
path follows the WAL record chain backwards from the divergence point.
The patch also has a fallback: if the record at the divergence point cannot
be read, it scans forward from the target control-file checkpoint, provided
that checkpoint precedes the divergence point.

The motivation is not to avoid reading target WAL. Both the current
implementation and the POC must read target WAL to build the page map.
The expected saving is avoiding redo of target WAL into relation files
that pg_rewind subsequently makes equal to the source, or removes because
they exist only on the target.

As a initial local benchmark, I compared current pg_rewind with this
POC. The workload generated approximately the following amount of
uncheckpointed target-side data:
current POC reduction
100 MB 1.404 s 0.979 s 30.3%
500 MB 5.358 s 3.991 s 25.5%
1000 MB 10.473 s 7.097 s 32.2%
2000 MB 49.904 s 31.294 s 37.3%
The result is a ~1.5x speedup in this workload, along with some tests
I did, and also make check-world passes.

would love to hear more thoughts on this.

--
Thanks :)
Srinath Reddy Sadipiralla
EDB: https://www.enterprisedb.com/