Let physical WAL senders restore missing segments from the archive

Hi hackers,

A standby can fall behind the WAL retained by its upstream while the
missing segments are still available in the archive. We would like to
keep that history in the archive, not retain all of it locally for a
replication slot. Giving the receiver its own restore_command is not
always an option either: we may be able to grant replication access
without giving it access to the archive. Could the sender provide the
missing WAL over the existing replication connection instead?

Fujii proposed this in 2010 [0], and Japin posted a patch in 2025 [1].
I would like to revisit it with the archive access restriction above as
the use case, and with private copies instead of restoring into the
normal WAL namespace.

Attached is a prototype. With wal_sender_restore = on, a physical sender
that cannot open a segment in pg_wal tries restore_command before
failing. This works on a primary or a [cascading] standby.

The temporary storage and cleanup questions came up when discussing
reviving lost replication slots [2]. Here each sender keeps at most one
restored segment in its own pg_wal/xlogtemp.*.walsender directory.

Alexander pointed out the cost of repeated failed restores on the
primary [3]. A request arriving within one second of a failed restore
waits for the remainder of that interval.

The existing timeline and flush/replay limits still apply. This does not
enable logical decoding from the archive or revive an invalidated slot.
It can serve a receiver without a slot, or with a usable physical slot.
Joao's proposal to revalidate physical slots [4] addresses that separate
part of the problem.

The prototype runs restore_command synchronously. A slow fetch prevents
keepalives and feedback processing, can cause receiver timeouts, and can
delay shutdown. An interruptible implementation would need to manage the
child command while servicing the replication connection.

What do you think?

Best regards, Andrey Borodin.

[0] https://www.postgresql.org/message-id/AANLkTinN=xsPOoaXzVFSp1OkfMDAB1f_d-F91xjEZDV8@mail.gmail.com
[1] https://www.postgresql.org/message-id/ME0P300MB0445C80039D6534452A87539B657A@ME0P300MB0445.AUSP300.PROD.OUTLOOK.COM
[2] https://www.postgresql.org/message-id/CALj2ACVUgCYVgKc42GDCFZo_=xjYyDtHXBhzyhT3s4UqAzLK4Q@mail.gmail.com
[3] https://www.postgresql.org/message-id/CAFh8B=nH=41scx3F_EAh2H5O1w-nj8b7uCMp_0z4p4wsv2tFDA@mail.gmail.com
[4] https://commitfest.postgresql.org/patch/6865/