| From: | Bharath Rupireddy <bharath(dot)rupireddyforpostgres(at)gmail(dot)com> |
|---|---|
| To: | Nathan Bossart <nathandbossart(at)gmail(dot)com> |
| Cc: | PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>, Álvaro Herrera <alvherre(at)kurilemu(dot)de> |
| Subject: | Re: REPACK (CONCURRENTLY) backend waits indefinitely when decoding worker fails to start |
| Date: | 2026-08-28 20:20:00 |
| Message-ID: | CALj2ACUpxBdkAfvfT4=DsQ4k5Faz7W9Sqk27j+Y4HWexPSvUdA@mail.gmail.com |
| Views: | Whole Thread | Raw Message | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-hackers |
Hi,
On Thu, Aug 27, 2026 at 9:47 AM Nathan Bossart <nathandbossart(at)gmail(dot)com> wrote:
>
> On Thu, Aug 27, 2026 at 09:41:49AM -0700, Bharath Rupireddy wrote:
> > REPACK (CONCURRENTLY) starts a decoding bgworker and then waits in
> > start_repack_decoding_worker() for the worker to set a shared-memory
> > flag. The wait has no liveness check on the worker itself. If the
> > worker never reaches that point (e.g., fork failure under memory
> > pressure, or when BecomeLockGroupMember() returns false, or early exit
> > before the shm_mq error redirect is set up), the backend waits
> > indefinitely with no way out other than cancellation. I reproduced
> > this with an induced fork failure, so I think we need to tighten this
> > for both PG19 and HEAD branches.
>
> Oops, I just concurrently reported this [0]. Note that teardown can
> deadlock, too.
>
> [0] https://postgr.es/m/apBpOVZOyqrakEr_%40nathan
Thanks. Here's my first attempt at fixing both the fork failure hang
and the teardown deadlock. I tried to use the parallel query approach
as much as possible.
The fork failure hang can occur because the backend running concurrent
repack sleeps on a CV and ignores the SIGUSR1 sent via bgw_notify_pid
by the postmaster upon fork failure.
The teardown deadlock can occur because the backend waits for the
worker to exit before detaching the error queue, while the worker is
blocked writing to a full queue, so both end up waiting on each other.
Although these issues seem rare to hit, I think it's good to tighten
the repack code because users can see them via SQL. Therefore, I think
we need to backpatch these to PG19. Please have a look at the attached
patch.
While here, I noticed that the same wait event is used for both the
worker startup wait and the file export wait. Ideally these would have
separate wait events, but given that the startup wait is typically
very short, reusing the same one seems fine.
--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com
| Attachment | Content-Type | Size |
|---|---|---|
| v1-0001-Fix-hang-and-deadlock-in-concurrent-REPACK-worker.patch | application/x-patch | 7.5 KB |
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Dmitry Dolgov | 2026-08-28 20:39:02 | Re: Randomize B-Tree page split location to avoid oscillating patterns |
| Previous Message | Zsolt Parragi | 2026-08-28 20:19:52 | Routed ON CONFLICT inserts broken by partition-local deferrable unique constraints in 19 and master |