Re: REPACK (CONCURRENTLY) can silently lose updates when the toast table is rewritten

From: shihao zhong <zhong950419(at)gmail(dot)com>
To: Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com>
Cc: Antonin Houska <ah(at)cybertec(dot)at>, Manu <manuelreyesbravo(at)gmail(dot)com>, Thom Brown <thom(at)linux(dot)com>, pgsql-hackers(at)lists(dot)postgresql(dot)org
Subject: Re: REPACK (CONCURRENTLY) can silently lose updates when the toast table is rewritten
Date: 2026-09-24 01:45:34
Message-ID: CAGRkXqR19T5MMoL=0-tuiYSO82uGfNUdGceSqoDZ9_Fvd6NtUw@mail.gmail.com
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hi Masahiko,

Thanks for reviewing it.

> I think we could error out if we detect a concurrent rewrite, so that
> users can re-run REPACK CONCURRENTLY. This check could also be done on
> the repack worker side

Agreed. v2 attached. REPACK now fails if the TOAST table was rewritten,
and the user can run it again.

The check stays in the backend, under the lock, though. If the worker
checks, a rewrite can still come after that check and before
copy_table_data() locks the TOAST table, and the update is lost the same
way. The backend takes the lock right after the worker is set up and
keeps it. Locking first would also work, but then the same race ends in
a deadlock instead of a clear error.

With the loop gone, the window Thom asked about is gone too. 0002 is the
test and is optional.

Shihao

On Wed, Sep 23, 2026 at 2:27 PM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com>
wrote:

> On Wed, Sep 23, 2026 at 9:23 AM Antonin Houska <ah(at)cybertec(dot)at> wrote:
> >
> > shihao zhong <zhong950419(at)gmail(dot)com> wrote:
> >
> > > > or whether the relfilenode should be re-checked after the snapshot
> is built
> > >
> > > Holding the toast lock from the start deadlocks. A session that asks
> for
> > > AccessExclusiveLock gets an XID before it waits, and the decoding
> worker
> > > waits for all XIDs while it sets up.
> >
> > The same (supposedly low) deadlock risk already exists for the main
> table, see
> > this comment in rebuild_relation():
> >
> > /*
> > * Start the worker that decodes data changes applied while we're
> > * copying the table contents.
> > *
> > * Note that the worker has to wait for all transactions with XID
> > * already assigned to finish. If some of those transactions is
> > * waiting for a lock conflicting with ShareUpdateExclusiveLock on
> our
> > * table (e.g. it runs CREATE INDEX), we can end up in a deadlock.
> > * Not sure this risk is worth unlocking/locking the table (and its
> > * clustering index) and checking again if it's still eligible for
> > * REPACK CONCURRENTLY.
> > */
> > start_repack_decoding_worker(tableOid);
> >
> > I'm not sure if locking the TOAST relation earlier would make the
> situation
> > worse.
>
> Agreed.
>
> So I think the simplest fix would be to acquire a lock on the TOAST
> table before starting the repack worker. It would make the case in
> question fail with a deadlock, instead of silently losing updates.
>
> The proposed patch also fixes the problem, but I'm concerned that it
> repeatedly starts and stops the repack worker without any limit. I
> think we could error out if we detect a concurrent rewrite, so that
> users can re-run REPACK CONCURRENTLY. This check could also be done on
> the repack worker side: after getting the relfilelocator of the TOAST
> table and initializing the logical decoding, the repack worker
> rechecks the relfilelocator. If they don't match, it raises an error.
>
> Regards,
>
> --
> Masahiko Sawada
> Amazon Web Services: https://aws.amazon.com
>

Attachment Content-Type Size
v2-0001-Fix-REPACK-CONCURRENTLY-losing-updates-after-a-TO.patch application/octet-stream 6.4 KB
v2-0002-Test-TOAST-rewrite-during-REPACK-CONCURRENTLY-sta.patch application/octet-stream 7.1 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message shihao zhong 2026-09-24 01:51:55 Re: Add a permission check to pg_stat_get_backend_subxact()
Previous Message shihao zhong 2026-09-24 01:30:19 Reset waitStart when a lock wait fails