Re: Assertion failure in update_local_synced_slot(), slotsync.c:364

From: Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com>
To: Zhijie Hou <houzhijie22(at)gmail(dot)com>
Cc: Amit Kapila <amit(dot)kapila16(at)gmail(dot)com>, Bertrand Drouvot <bertranddrouvot(dot)pg(at)gmail(dot)com>, PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Assertion failure in update_local_synced_slot(), slotsync.c:364
Date: 2026-11-17 18:26:24
Message-ID: CAD21AoD2naqObw2Qjo9UdfnAOdJ5ecdBn+Zb9w4aNi9J_r8bZQ@mail.gmail.com
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On Mon, Nov 16, 2026 at 12:09 AM Zhijie Hou <houzhijie22(at)gmail(dot)com> wrote:
>
> Hi,
>
> On Mon, Nov16, 2026 at 02:35 PM Masahiko Sawada wrote:
> > On Sun, Nov 15, 2026 at 8:35 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> > >
> > > On Mon, Nov 16, 2026 at 9:50 AM Zhijie Hou <houzhijie22(at)gmail(dot)com> wrote:
> > > >
> > > > On Sat, Nov14, 2026 at 02:18 PM Masahiko Sawada wrote:
> > > > > On Fri, Nov 13, 2026 at 7:35 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> > > > >
> > > > > While I agree that retaining the previous reason doesn't make us
> > > > > persist the slot, I think it's safe only because the previous
> > > > > reason is never SS_SKIP_NONE for a temporary slot at that point.
> > > > > That's true today, but it depends on several things and it would
> > > > > be hard to modify these code correctly in the future.
> > > > >
> > > > > I think the root cause is that commit 788ec96d591 replaced
> > > > > found_consistent_snapshot with the check of
> > > > > slotsync_skip_reason, but they're not equivalent:
> > > > > found_consistent_snapshot is false unless the current cycle
> > > > > synced the slot and found a consistent snapshot, whereas
> > > > > slotsync_skip_reason is SS_SKIP_NONE unless the current cycle
> > > > > detected a reason to skip. So an alternative idea would be to
> > > > > bring back found_consistent_snapshot part from the commit. That
> > > > > way, the decision of persisting the slot doesn't depend on the
> > > > > skip_reasons at all.
> > > > >
> > > > > As for the reported reason, a temporary slot is the one we have
> > > > > not confirmed to reach a consistent point yet. So I think we can
> > > > > simply report no_consistent_snapshot for a temporary slot if the
> > > > > current cycle didn't find a consistent snapshot, without relying
> > > > > on what happened in the previous cycles.
> > > >
> > > > I think it would be unnecessary to reintroduce the
> > > > found_consistent_snapshot out-parameter. Because by definition,
> > > > slotsync_skip_reason is SS_SKIP_NONE (NULL in
> > > > pg_replication_slots) only when slot synchronization succeeds.
> > > > Even before commit 788ec96d591 (since commit e68b6adad9),
> > > > resetting slotsync_skip_reason to SS_SKIP_NONE on a no-op cycle
> > > > for a temporary slot was already a bug, because synchronization of
> > > > that temporary slot had not succeeded yet.
> > > >
> > >
> > > I agree that the bug is because we didn't maintain/update
> > > slotsync_skip_reason correctly and we anyway need to maintain that
> > > state to correctly display it in pg_replication_slots. Also, the
> > > slot already carries the information about consistent_snapshot.
> > > SS_SKIP_NO_CONSISTENT_SNAPSHOT says the slot hasn't reached a
> > > consistent point, and it is visible in pg_replication_slots. A flag
> > > passed back to the caller to say the same thing would be a duplicate
> > > of that state.
> >
> > I agree that the v2 patch fixes the problem correctly. My concern is
> > rather about readability and maintainability, which is why I proposed
> > the PoC patch.
> >
> > + else if (slot->data.persistency == RS_TEMPORARY)
> > + {
> > + /*
> > + * We reach this point when a previous cycle updated confirmed_flush
> > + * but failed to build a consistent snapshot, and the remote slot has
> > + * not advanced beyond the local slot since then.
> > + */
> > + skip_reason = SS_SKIP_NO_CONSISTENT_SNAPSHOT;
> > + }
> >
> > Some my concerns are:
> >
> > - When a temporary slot finds a consistent snapshot, it should be
> > persisted before update_local_synced_slot() is called for the slot
> > next time. If update_local_synced_slot() is called twice for a
> > temporary slot without the remote slot advancing, the two calls give
> > different results. The first call syncs the slot, finds a consistent
> > snapshot and leaves SS_SKIP_NONE, but the second call enters the new
> > branch and sets SS_SKIP_NO_CONSISTENT_SNAPSHOT, although nothing has
> > changed since the first call. This doesn't happen today only because
> > the caller persists the slot right after the first call.
>
> To make this invariant explicit and catch any future caller changes, I
> think we can add an assertion at the entry of update_local_synced_slot()
> verifying that a temporary slot with a valid confirmed_flush always has
> slotsync_skip_reason != SS_SKIP_NONE. (See the updated patch for details.)
>
> I still prefer this over reintroducing an out-function parameter, since
> update_local_synced_slot() already has to maintain slotsync_skip_reason
> accurately on every path for pg_replication_slots. Relying on
> slot->slotsync_skip_reason != SS_SKIP_NONE in
> update_and_persist_local_synced_slot() avoids maintaining and passing a
> duplicate state for the same condition which I think that's worth keeping.
>
> > - The comment explains the branch by what happened in a previous
> > cycle. So one needs to go through the possible history of the slot to
> > see why it's correct, whereas what the code checks is just that the
> > temporary slot hasn't found a consistent snapshot in this call.
> >
> > - We set SS_SKIP_NO_CONSISTENT_SNAPSHOT in two places for what is
> > actually the same condition, which makes the code harder to follow.
>
> To address these points, I moved found_consistent_snapshot to the outer
> scope of update_local_synced_slot() and checking "if
> (slot->data.persistency == RS_TEMPORARY && !found_consistent_snapshot)"
> in one central place, with the comment updated to explain the check in
> terms of the current call. This also emits the LOG message in both
> cases, addressing Amit's point upthread.
>
> >
> > I'm concerned that not only changes to this function but also changes
> > to its callers could break these assumptions, and nothing near those
> > places would tell us about them.
>
> Attached is the v3 patch set (with the TAP test split into 0002). Please
> let me know what you think.

Thank you for updating the patch!

The v3 patch looks good to me. It changes more code than v2, but I
prefer this version.

What I cared about is that this function is easy to follow so that we
don't get a similar bug again. This bug was introduced by the
refactoring in 788ec96d591, and we modified the same part of
update_local_synced_slot() in 85c17f612af two months later to handle
no-op sync cycles, but nobody noticed the bug. Also, we already had
the assertion that the slot is temporary if it cannot find a
consistent snapshot, but it didn't help because no test reached that
state. So I don't think we can rely much on assertions here. In v3, we
set SS_SKIP_NO_CONSISTENT_SNAPSHOT in one place based only on the
result of the current call, and the new assertion is checked whenever
we sync a temporary slot, so a change that breaks the assumption would
be caught by existing tests.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Haibo Yan 2026-11-17 18:27:10 Re: [PATCH v1] pg_stash_advice: Reject embedded NUL bytes in persistence files
Previous Message Alexander Pyhalov 2026-11-17 18:07:14 CachedPlan Invalidation in SQL function