[Patch] Batch fsyncs when recycling WAL segments

Hi,

While working on [1], I read on the "Pre-allocating WAL files" thread [2]
that the fsyncs we do when recycling WAL segments are a real bottleneck,
and Andres suggested batching them: rename a batch first, then fsync the
files and pg_wal once, instead of the current per-segment
durable_rename() in InstallXLogFileSegment(). The attached patch does
that. (Andres and Nathan, who were on that thread, are cc'd.)

The tricky part was durability. As soon as a segment is renamed it can
be handed to the WAL write path, but the rename isn't on disk yet, and
issue_xlog_fsync() only fsyncs the segment file, never the directory.
So if a backend writes a commit into a just-recycled segment and we
crash before the batched fsync, the rename can be lost.

What I ended up with is a small "durability frontier" in XLogCtl (a
timeline and the highest segment whose rename is known durable). The
checkpointer advances it after the batched fsync, and XLogFileInit()
checks it before letting anyone write into a segment; if the segment
isn't covered yet it just fsyncs the file and pg_wal itself. Normally
the checkpointer stays ahead, so that fallback basically never fires(?)
it's
only there to close the crash window. An injection point and a TAP test
force that window open and confirm a committed row survives an immediate
crash.

Rough numbers: a WAL-heavy workload recycling ~44 segments per
checkpoint (min_wal_size=1GB, max_wal_size=2GB, wal_recycle=on,
fsync=on), ext4 on an Azure disk, 14 checkpoints with the first dropped:

median mean min max
master 138.6 ms 143 ms 123 ms 171 ms
patch 27.3 ms 28 ms 25 ms 39 ms

~5x here, and a lot steadier: master pays ~44 flushes every checkpoint,
the patch pays one.

Is the frontier + write-path check the right shape for this, or is there a
simpler invariant that keeps the write path from ever reaching a
not-yet-durable recycled segment?

Thoughts?

Regards,
Ayush

[1]
https://www.postgresql.org/message-id/flat/CAJTYsWXAPabSWBeT=bqJD+6w0D4-ozvoVoH7KiFUncM88bsQOg(at)mail(dot)gmail(dot)com
[2]
https://www.postgresql.org/message-id/745zvagaf6tfn2zbvtmqvxsr6kmybjbzcw6rsm55jnql4233je%40tudeipoewruo