Re: [Patch] Batch fsyncs when recycling WAL segments

Hi,

I had another pass at this and have attached v3, now split into two patches.
The removal and recycling cases have one important difference: only recycling
exposes a new WAL filename to the write path.

0001 is the straightforward bit. For segments that a checkpoint removes, it
does the unlinks first, fsyncs pg_wal once, and then cleans up the archive
status files. I left the timeline-switch path alone. This is not only for
wal_recycle=off; with recycling enabled, surplus segments still take the
removal path.

I reran the removal benchmark with the default 16 MiB WAL segment size. The
numbers below are for 0001 alone; 0002 was not applied. This was on ext4 with
fsync enabled and wal_recycle=off, so every eligible segment went through the
removal path. Each number is the median of five alternating runs after three
warmups:

files removed master 0001
1 22 ms 23 ms
8 53 ms 32 ms
32 135 ms 64 ms
64 201 ms 97 ms
128 394 ms 154 ms

The one-file case was within run-to-run variation, as expected. Once there
are several files to remove, the per-file directory fsyncs start to show up;
at 128 files the time went from 394 ms to 154 ms, about 60% lower (2.52x).
This is not a TPS result, just checkpoint cleanup time. Since 0001 avoids
the write-path changes in 0002, I thought it was worth splitting out for
separate consideration.

0002 is the awkward bit, and I have left it marked WIP. Going back over v2,
I found a hole in the durable-segment high-water mark. A foreground process
can find the names in the pending recycle batch already occupied, install its
own segment at a higher number, and move the frontier past lower renames that
are not durable yet.

The current patch plugs that hole with an active-batch marker. While a batch
is in flight, XLogFileInit() takes the file-and-directory fsync fallback even
if the frontier says the segment is covered. The new TAP test forces exactly
this interleaving; without the marker check, it times out waiting for the
fallback.

That handles the case I found, but I am still on the fence about 0002. It
feels like a fair amount of plumbing between checkpoint cleanup and the WAL
write path just to save these fsyncs. Is there a cleaner invariant that I am
missing, or a less invasive way to make the batched renames safe?

Regards,
Ayush