Making WAL archiving faster — multi-file support and async ideas

Hi hackers,

We’ve been thinking about how to make WAL archiving faster.

This topic was previously discussed in [1], and we’ve taken a first step by
implementing the attached patch, which adds support for archiving multiple
WAL files in one go.

The idea is straightforward: instead of invoking the archive command or
callback once per WAL file, we allow passing a batch of files. The patch
introduces support for new placeholders:

-

%F – list of WAL file names
-

%P – list of their full paths
-

%N – number of files

Since PostgreSQL already reads multiple files into memory and caches them
before archiving, this change avoids repeated fork() calls and reduces
overhead in high-throughput setups.

Of course, there are trade-offs. After discussing with Andrey Borodin, we
noted that if even one file in the batch fails to archive, we currently
have to retry the whole batch. While it’s technically possible to return a
list of successfully archived files, that would complicate the API and
introduce messy edge cases.

So we’re also exploring a more flexible idea: an asynchronous archiver mode.

The idea is to have PostgreSQL write WAL file names (marked .ready) into a
FIFO or pipe, and let an archive process or library asynchronously consume
and archive them. It would send back confirmations (or failures) through
another pipe, allowing PostgreSQL to retry failed files as needed. This
could decouple archiving from the archiver loop and open the door to more
efficient and parallel implementations.

We’d appreciate feedback on both directions:

-

Do you think the idea in the attached patch — batching WAL files for
archiving — is viable? Is it something worth pursuing?
-

What do you think about the async archiver concept? Would it fit
PostgreSQL’s architecture and operational expectations?

Thanks,
Stepan Neretin

[1]
https://www.postgresql.org/message-id/flat/BC335D75-105B-403F-9473-976C8BBC32E3%40yandex-team.ru#d45caa9d1075734567164f73371baf00