Re: BUG #19557: Parallel leader stuckat ParallelFinish leaked InterruptHoldoffCount

Adding Robert and Andres because they know these areas best and can provide
some valuable insights to this issue.

On Fri, 17 Jul 2026 at 22:00, PG Bug reporting form <noreply(at)postgresql(dot)org>
wrote:

> The following bug has been logged on the website:
>
> Bug reference: 19557
> Logged by: Jacob Mørk Brejnbjerg
> Email address: jacob(at)brejnbjerg(dot)com
> PostgreSQL version: 18.4
> Operating system: Ubuntu 24.04 (kernel 6.8.0-124-generic)
> Description:
>
> A backend executing a parallel query has been stuck for 40+ hours in
> wait_event IPC/ParallelFinish. All of its parallel workers exited cleanly
> (exit code 0, nothing logged). pg_cancel_backend() and
> pg_terminate_backend() have no effect. We kept the backend alive and
> debugged it with gdb + debug symbols; the evidence shows a leaked
> InterruptHoldoffCount, plus a design interaction that turns any such leak
> into a permanently unkillable backend.
>
> Possibly relevant configuration: io_method = io_uring,
> max_parallel_workers_per_gather = 48,
> shared_preload_libraries = 'timescaledb,pg_parquet'
> (timescaledb 2.27.2, pg_parquet 0.5.1).
> PostgreSQL 18.4 (Ubuntu 18.4-1.pgdg24.04+1).
>
> The query is a COPY (SELECT <aggregation over one table> GROUP BY ...)
> TO STDOUT (FORMAT parquet) issued via pg_parquet's COPY hook; the inner
> select runs a parallel plan (Gather, 10+ workers planned) over
> TimescaleDB compressed-hypertable chunks, i.e. an AIO-read-heavy scan.
>
> Backtrace of the stuck leader (main thread; the only other thread is an
> io_uring kernel worker, "iou-wrk-<pid>"):
>
> #0 epoll_wait (epfd=754, events=..., maxevents=1, timeout=-1)
> #1 WaitEventSetWait ()
> #2 WaitLatch ()
> #3 WaitForParallelWorkersToFinish ()
> #4 ExecParallelFinish ()
> #5-#8 (Gather shutdown path, static functions)
> #9 standard_ExecutorRun ()
> #10-#11 PortalRun ()
> #12-#17 pg_parquet.so (COPY TO hook driving the portal)
> #18 prev_ProcessUtility (timescaledb process_utility.c:124)
> #19 timescaledb_ddl_command_start (...)
> #20-#22 PortalRun ()
> #23-#24 PostgresMain ()
>
> Key state read from the live process via gdb:
>
> InterruptPending = 1
> ProcDiePending = 1 <- pg_terminate_backend arrived, never
> serviced
> QueryCancelPending = 1 <- pg_cancel_backend arrived, never serviced
> InterruptHoldoffCount = 1 <- held, forever
> CritSectionCount = 0
> num_held_lwlocks = 0 <- NO LWLock is held
> ParallelMessagePending = 1 <- worker messages queued, unprocessable
>
> /proc signal state: SigPnd/ShdPnd are all zero - the SIGTERM/SIGINT were
> delivered and consumed by the handlers (which set the flags above); the
> backend wakes from WaitLatch, but CHECK_FOR_INTERRUPTS() is a no-op with
> InterruptHoldoffCount > 0, so it just re-sleeps.
>
> Why this is a permanent hang and not just an uncancellable wait:
> worker-exit accounting is itself interrupt-driven. The workers' detach
> notifications are only processed via CHECK_FOR_INTERRUPTS() ->
> ProcessParallelMessages(), which is what clears each worker's error_mqh.
> With the holdoff leaked, ProcessParallelMessages never runs
> (ParallelMessagePending stays 1 - verified in the live process), so
> WaitForParallelWorkersToFinish() believes the workers are still attached
> and waits forever. The same leaked holdoff blocks ProcDiePending. One
> leaked holdoff therefore produces both symptoms; only SIGKILL
> (crash-restart) or a debugger can clear the backend.
>
> On the origin of the leak: num_held_lwlocks = 0 rules out a stuck
> LWLockAcquire, and the server log shows NO error from this backend for
> its entire lifetime, which we believe rules out the
> "ereport(ERROR) raised under HOLD, caught by PG_TRY" pattern (the raise
> would have been logged by errfinish). That points to a straight-line
> code path that called HOLD_INTERRUPTS() and returned without
> RESUME_INTERRUPTS(). We audited the read path: bufmgr.c, read_stream.c,
> shm_mq.c, condition_variable.c contain no bare holds; timescaledb 2.27.2
> and pg_parquet 0.5.1 (and pgrx 0.16.1) contain no HOLD_INTERRUPTS at
> all. The remaining bare holds on this workload's path are in
> src/backend/storage/aio/aio.c (four HOLD/RESUME regions, including
> pgaio_io_reclaim which runs completion callbacks under HOLD) and the
> parallel-message path itself. Given io_method = io_uring and an
> AIO-heavy scan we suspect an AIO path, but we could not identify the
> exact site from the live process.
>
> Frequency: once in ~6 days of heavy production use of this COPY path
> (thousands of executions/day). Not deterministically reproducible so
> far.
>
> We have a gcore of the stuck backend and can keep it (and/or the live
> process) available - happy to run additional gdb commands against it.
>
> Separately, it may be worth considering a defensive measure: because
> WaitForParallelWorkersToFinish depends on interrupt processing for its
> own termination condition, any holdoff leak from any subsystem converts
> into a silent, unkillable backend. Detecting "waiting on ParallelFinish
> with INTERRUPTS_CAN_BE_PROCESSED() false and ParallelMessagePending set"
> and at least logging (or treating it as an invariant violation) would
> make this failure mode visible instead of silent.
>
>
>
>
>

--
Regards,
Rafia Sabih
CYBERTEC PostgreSQL International GmbH