Re: Let physical WAL senders restore missing segments from the archive

On Thu, 22 Oct 2026 at 21:43, Andrey Borodin <x4mmm(at)yandex-team(dot)ru> wrote:
>
> Hi hackers,
>
> A standby can fall behind the WAL retained by its upstream while the
> missing segments are still available in the archive. We would like to
> keep that history in the archive, not retain all of it locally for a
> replication slot. Giving the receiver its own restore_command is not
> always an option either: we may be able to grant replication access
> without giving it access to the archive. Could the sender provide the
> missing WAL over the existing replication connection instead?
>
> Fujii proposed this in 2010 [0], and Japin posted a patch in 2025 [1].
> I would like to revisit it with the archive access restriction above as
> the use case, and with private copies instead of restoring into the
> normal WAL namespace.
>
> Attached is a prototype. With wal_sender_restore = on, a physical sender
> that cannot open a segment in pg_wal tries restore_command before
> failing. This works on a primary or a [cascading] standby.
>
> The temporary storage and cleanup questions came up when discussing
> reviving lost replication slots [2]. Here each sender keeps at most one
> restored segment in its own pg_wal/xlogtemp.*.walsender directory.
>
> Alexander pointed out the cost of repeated failed restores on the
> primary [3]. A request arriving within one second of a failed restore
> waits for the remainder of that interval.
>
> The existing timeline and flush/replay limits still apply. This does not
> enable logical decoding from the archive or revive an invalidated slot.
> It can serve a receiver without a slot, or with a usable physical slot.
> Joao's proposal to revalidate physical slots [4] addresses that separate
> part of the problem.
>
> The prototype runs restore_command synchronously. A slow fetch prevents
> keepalives and feedback processing, can cause receiver timeouts, and can
> delay shutdown. An interruptible implementation would need to manage the
> child command while servicing the replication connection.
>
> What do you think?
>
>
> Best regards, Andrey Borodin.
>
> [0] https://www.postgresql.org/message-id/AANLkTinN=xsPOoaXzVFSp1OkfMDAB1f_d-F91xjEZDV8@mail.gmail.com
> [1] https://www.postgresql.org/message-id/ME0P300MB0445C80039D6534452A87539B657A@ME0P300MB0445.AUSP300.PROD.OUTLOOK.COM
> [2] https://www.postgresql.org/message-id/CALj2ACVUgCYVgKc42GDCFZo_=xjYyDtHXBhzyhT3s4UqAzLK4Q@mail.gmail.com
> [3] https://www.postgresql.org/message-id/CAFh8B=nH=41scx3F_EAh2H5O1w-nj8b7uCMp_0z4p4wsv2tFDA@mail.gmail.com
> [4] https://commitfest.postgresql.org/patch/6865/

Hi!
Reviewed patch.
There are issues with test races, and consequent promotions , but they
are already fixed in [0]. I briefly looked at them - should be OK.
Will wait for list posting.
There are additional improvements possible.

1. TIMELINE_HISTORY support. If the replica contains only the current
timeline history file, say X, the replication proto client can request
TIMELINE_HISTORY X - 1 and fail. attached 0001 implements that. This
is a separate improvement though and I do not intend to merge this in
one commit with the main feature.

2.WALbuffers WAL read can fail due to on-disk xlog files removal.
Being discovered while reviewing this, this is a separate issue, so I
reported it as [1].

===

Some nits, in unsorted level of severity

Also, this magic number

> /* Minimum delay after a failed archive restore, in milliseconds. */
> #define WALSND_RESTORE_DELAY 1000

Should we make this a GUC?

> /* Time of last archive restore failure, protected by restore_mutex. */
> slock_t restore_mutex;
> TimestampTz lastRestoreFailure;

Can we use a single atomic integer?

[0] https://github.com/x4m/postgres_g/commits/refs/heads/walsender-archive-v1-followups/
[1] https://www.postgresql.org/message-id/CALdSSPid5M%3DqE9UHz5duQE_%2BxJF0bn1OQE4W8fxb3%2BBtZ0Yk7w%40mail.gmail.com

--
Best regards,
Kirill Reshke