Re: [Patch] Batch fsyncs when recycling WAL segments

Hi,

On Thu, 3 Sept 2026 at 00:27, Ayush Tiwari <ayushtiwari(dot)slg01(at)gmail(dot)com> wrote:
>
> I had another pass at this and have attached v3, now split into two patches.
> The removal and recycling cases have one important difference: only recycling
> exposes a new WAL filename to the write path.
>
> 0001 is the straightforward bit. For segments that a checkpoint removes, it
> does the unlinks first, fsyncs pg_wal once, and then cleans up the archive
> status files. I left the timeline-switch path alone. This is not only for
> wal_recycle=off; with recycling enabled, surplus segments still take the
> removal path.
>
> I reran the removal benchmark with the default 16 MiB WAL segment size. The
> numbers below are for 0001 alone; 0002 was not applied. This was on ext4 with
> fsync enabled and wal_recycle=off, so every eligible segment went through the
> removal path. Each number is the median of five alternating runs after three
> warmups:
>
> files removed master 0001
> 1 22 ms 23 ms
> 8 53 ms 32 ms
> 32 135 ms 64 ms
> 64 201 ms 97 ms
> 128 394 ms 154 ms
>
> 0002 is the awkward bit, and I have left it marked WIP. Going back over v2,
> I found a hole in the durable-segment high-water mark. A foreground process
> can find the names in the pending recycle batch already occupied, install its
> own segment at a higher number, and move the frontier past lower renames that
> are not durable yet.
>
> The current patch plugs that hole with an active-batch marker. While a batch
> is in flight, XLogFileInit() takes the file-and-directory fsync fallback even
> if the frontier says the segment is covered. The new TAP test forces exactly
> this interleaving; without the marker check, it times out waiting for the
> fallback.
>
> That handles the case I found, but I am still on the fence about 0002. It
> feels like a fair amount of plumbing between checkpoint cleanup and the WAL
> write path just to save these fsyncs. Is there a cleaner invariant that I am
> missing, or a less invasive way to make the batched renames safe?

Attached v4, rebased on master.

I noticed I'd made directory-sync failures PANIC by default in v3.
0001 now keeps the old LOG-level behavior and leaves archive status
alone if the batch sync fails.

0002 is just rebased, with its TAP test renumbered. I've kept it WIP
and added a nocfbot prefix pending discussion.

Regards,
Ayush