Re: Introduce XID age based replication slot invalidation

Hi,

On Mon, Aug 31, 2026 at 7:35 AM Nisha Moond <nisha(dot)moond412(at)gmail(dot)com> wrote:
>
> > > AFAICS this can also invalidate the reserved pg_conflict_detection slot. The
> > > checkpoint terminates its logical replication launcher, but the restarted
> > > launcher reacquires the invalid slot while subscriptions and workers still
> > > consider conflict information retention active.
>
> > Since the pg_conflict_detection slot is internally created and
> > maintained for the user, I don't think invalidating it (by terminating
> > the launcher and asking every apply worker to stop conflict retention)
> > is the right approach. After invalidating, the slot cannot be dropped,
> > and the conflict detection feature will no longer be working
> > correctly, and the launcher will not error out for invalidated slots.
> > The user doesn't even have an easy way to deal with this situation.
> > They would have already set retain_dead_tuples and
> > max_retention_duration for the subscription, but they now have no way
> > to get conflict detection to work again.
> >
> > That said, one of the design goals for the XID-age based slot
> > invalidation feature is that no slot blocks vacuum indefinitely.
> >
> > My proposal is to check the age of each worker's
> > oldest_nonremovable_xid against the XID-age GUC in each apply worker
> > in should_stop_conflict_info_retention(). If it is aged, stop the
> > retention. This mechanism is simple to reason about and fits well with
> > the backoff we already have with max_retention_duration. This keeps
> > the slot valid but backs off from holding the vacuum.
>
> The approach looks good to me.
>
> My concern is that the launcher freezes the slot's xmin based on
> policy, independently of the worker's state. Because of this, a
> worker-side check cannot really put a limit on how long the xmin can
> be held.
>
> Launcher only advances the slot's xmin when all retention-active
> subscriptions are enabled and their workers are running. This is
> intentional; otherwise, a disabled subscription could miss conflicts
> when it comes back.
>
> This leads to two cases:
> 1) If a subscription is disabled, or its worker has crashed/cannot be
> started, there is no worker to call
> should_stop_conflict_info_retention(). Nothing else would release the
> old xmin.

Right. But one can set the retention duration to a really long value
and disable subscriptions even today to block the slot's xmin and
vacuum. Although we documented this and accepted it as a behavior the
user needs to be aware of, in the launcher we don't do anything about
it. An idea here could be having the launcher detect such
subscriptions with no workers (i.e. disabled), or crashing apply
workers, or initial table syncs taking a long time, or workers
applying large in-progress transactions, and having its own retention
duration check to back off or inform the workers that their retention
duration has passed because they didn't show up in a while. I haven't
looked at the conflict detection feature discussion thread in depth,
but I guess this would have been discussed there.

> 2) Even if the worker is running and caught up, its
> oldest_nonremovable_xid can be much newer than the slot's frozen
> data.xmin. In that case, the worker sees a relatively young XID and
> does not back off, while the slot continues to hold the older xmin.

That is correct. Because the slot's xmin is the minimum of each
worker's oldest xid (oldest_nonremovable_xid) and computed by the
launcher, yes that is possible. So I agree with the point below on
also driving the xid age check and backoff from the launcher, in
addition to each worker.

> So, if we exempt the slot and rely on the backoff, then in these cases
> the slot stays valid, but max_slot_xid_age stops being a bound on how
> long
> its xmin is held.
>
> One idea would be to drive the check from the launcher rather than the
> apply worker. It holds the slot, so it can compare the slot's actual
> data.xmin
> and can act for subscriptions that have no live worker. A worker-side
> check would still be worth having on top of that as a refinement,
> letting one lagging subscription back off while the others keep
> retaining.
>
> Thoughts?

Thanks a lot for the review. I'm still getting familiar with the
conflict detection feature, so please let me know if I'm missing
anything in the details below.

Here are two approaches.

One approach is to never invalidate the slot, and to honour
max_slot_xid_age by stopping conflict info retention for whoever is
holding the slot's xmin back. On the worker side, the existing
should_stop_conflict_info_retention() stops retention when the
worker's oldest xid has aged past the age limit, whether or not a
retention duration is set. On the launcher side, once per cycle, right
after it tries to advance the xmin as usual, if the xmin has aged past
the age limit it stops retention by setting subretentionactive to
false in pg_subscription for the disabled subscriptions, the
subscriptions with no apply worker running, and the subscriptions
whose apply worker has no valid oldest xid yet or whose oldest xid
still equals the slot's xmin. Workers whose oldest xid has moved past
the xmin go on retaining. Note that a worker's oldest xid can never be
older than the slot's xmin, since it starts at the xmin and only moves
forward, so the workers holding the xmin back are exactly the ones
whose oldest xid still equals it.

Once those subscriptions stop retaining they no longer count towards
the xmin, so it advances on the next cycle, or goes invalid if nothing
retains any more, and vacuum is unblocked. Resume needs no changes, a
stopped subscription comes back the same way it does after a
max_retention_duration stop. So the slot's xmin cannot stay past the
age limit for more than a launcher cycle or two.

An alternative approach is to invalidate the slot and automatically
drop it in the launcher. Currently the launcher drops the slot only
when no subscription has retain_dead_tuples enabled any more. Apply
workers may still be running then, but none of them is maintaining an
oldest xid, so dropping there is safe and easy. However, invalidation
terminates the launcher since it holds the slot, so the drop and
recreate has to happen in the restarted launcher, which comes up
holding a slot that is now invalid while the apply workers carry on
with retention active and their oldest xids derived from the old slot.
The moment the slot is invalidated nothing holds the xmin any more, so
vacuum can remove dead tuples the workers still count on, and a
recreated slot starts at a newer xmin than the workers' oldest xids.
That means the drop and recreate has to be coordinated with the
workers, they need to stop and restart so their oldest xids are
computed afresh from the new slot, and the retention state in
pg_subscription has to be kept in step with the slot's lifecycle
across all of this.

I prefer the first approach, since it stops retention only for the
subscriptions holding the xmin back, they resume on their own once
caught up, and the other subscriptions keep detecting conflicts
throughout. The second stops conflict detection more broadly and needs
coordination with all the apply workers.

Thoughts?

Thanks for reading this far!

--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com