Re: Parallel heap vacuum

Hi Bharath,

I have a question about the interaction between parallel heap scanning and
the wraparound failsafe in v21.

heap_vacuum_rel() calls lazy_check_wraparound_failsafe() before
dead_items_alloc(). If that check activates
the failsafe, parallel_lazy_scan_init() returns immediately, so the heap
scan runs serially.

However, if the failsafe activates after parallel scanning has started, the
existing workers continue
scanning and freezing, picking up the shared failsafe flag and disabling
cost-based delays and the buffer
access strategy.

Is there a reason to avoid starting parallel heap workers when the failsafe
is already active?
For large tables parallel freezing could be particularly useful.

Thanks!

Regards,
Vadim Ponomarev

чт, 12 нояб. 2026 г. в 00:34, Bharath Rupireddy <
bharath(dot)rupireddyforpostgres(at)gmail(dot)com>:

> Hi,
>
> On Fri, Jul 17, 2026 at 10:03 AM Bharath Rupireddy
> <bharath(dot)rupireddyforpostgres(at)gmail(dot)com> wrote:
> >
> > The main changes in this version:
> >
> > 1/ Moved the parallel-vacuum callbacks out of the table AM interface.
> > As suggested upthread, the access method now passes them to
> > vacuumparallel.c through a local callback struct, so the generic
> > parallel vacuum code no longer calls into the table AM.
>
> I have been working on this feature for a while now and discussing the
> design with Sawada-san off-list.
>
> A problem with the v20 patch's callback approach is that it requires
> table AM implementers wanting to reuse the code to register their
> parallel callbacks, which means the table AM shared library preload
> and a restart. This doesn't sound like good interfacing.
>
> So, keeping the design goal of reusing the parallel code from
> vacuumparallel.c as much as possible, while keeping the layering
> intact (vacuumlazy.c calling into vacuumparallel.c and not
> vice-versa), I came up with the attached v21 patches. The idea is to
> implement the parallel heap scan in the heap AM layer, i.e. in
> vacuumlazy.c itself.
>
> Please have a look at the attached 3 patches. 0001 moving the reusable
> code to separate functions and structures, 0002 moving the common heap
> scan state out of LVRelState, and 0003 implementing the parallel heap
> scan (i.e. phase 1) for vacuum.
>
> I benchmarked it (attached script) on an i4i.4xlarge EC2 instance with
> local NVMe storage with a best case [1] to understand the
> characteristics and contention points better. The patch makes the heap
> scan 2.4x to 4.3x faster on the 10GB table (CPU bound) and 1.5x to
> 3.6x faster on the 50GB one (storage bound) with 2 to 8 workers, which
> makes vacuum 1.1x to 1.5x faster on a table with indexes and up to
> 4.3x faster on one without.
>
> I added some design notes in 0003's commit message and listed the ones
> for some thoughts below:
>
> 1/ When the dataset fits in memory (s_b and RAM), the heap scan is CPU
> bound and gains more. When it doesn't, it becomes storage bound and
> gains less. With the default 2MB ring buffer, vacuum has to wait for
> its own dirty buffers to be written out.
>
> 2/ Chunk size, the number of heap blocks each worker claims in one go,
> causes VM page contention. A chunk size of 32K blocks, what a single
> VM page covers, gains more, while 1K gives only up to 2.3x at 8
> workers on the 10GB table when CPU bound. Since I used a chunk as the
> eager scan region and as the point where a process checks the dead TID
> store memory limit, I kept it fixed, so both stay the same size
> through the scan.
>
> 3/ Phase 2 uses parallel workers and phase 3 stays serial, so the gain
> in vacuum time is smaller, since phase 1 is only 25% to 50% of the
> whole. With 4 indexes and 8 workers on the 10GB table, phase 1 is 43%
> of the VACUUM time, so a 4x heap scan gain shows up as 1.4x overall.
>
> 4/ The cost of shared dead TID store lookups vs. a local store can be
> reduced by caching the last block entry looked up in
> TidStoreIsMember(). I will discuss it in a separate thread.
>
> 5/ The cost of the single exclusive lock per heap page on the whole
> radix tree for the shared dead TID store is 11% of phase 1 at 4
> workers and 17% at 8 workers on the 10GB table. I don't think we
> benefit much from introducing radix tree optimizations for concurrent
> writers such as OLC or ROWEX. I read the ART papers, and even if we
> were to implement one, OLC seems to be enough to reduce write lock
> contention, since vacuum doesn't have concurrent writers and readers
> of the radix tree at the same time (phase 1 all writers, phases 2 and
> 3 all readers). ROWEX helps to minimize reader restarts when writers
> are modifying concurrently, which may not be applicable as far as
> vacuum is concerned.
>
> Appreciate thoughts. Thanks.
>
> [1] Test setup: i4i.4xlarge, 16 vCPU, 128GB RAM, local NVMe storage.
> Unlogged table, 1 int column, 226 rows per page, 20% of rows deleted.
> Each btree index is 6.2GB at 10GB and 31GB at 50GB. s_b 32GB, m_w_m
> 1GB, ring buffer 2MB, 32768-block chunk. Avg of 3 runs.
>
> ```
> Phase 1, heap scan, HEAD -> patched (gain)
> table indexes 0 workers 2 workers 4 workers
> 8 workers
> 10GB 0 12.1 -> 12.2 (1x) 12.1 -> 5.0 (2.4x) 12.1 ->
> 3.7 (3.3x) 12.2 -> 2.8 (4.3x)
> 10GB 1 12.2 -> 12.1 (1x) 12.0 -> 5.1 (2.4x) 12.0 ->
> 3.7 (3.2x) 12.2 -> 3.0 (4.0x)
> 10GB 2 12.0 -> 12.0 (1x) 12.2 -> 5.1 (2.4x) 12.2 ->
> 3.7 (3.3x) 12.2 -> 3.0 (4.0x)
> 10GB 4 12.0 -> 12.2 (1x) 12.3 -> 5.1 (2.4x) 12.3 ->
> 3.7 (3.3x) 12.3 -> 3.1 (4.0x)
> 50GB 0 60.3 -> 60.7 (1x) 60.3 -> 23.6 (2.6x) 60.5 ->
> 17.8 (3.4x) 60.8 -> 17.1 (3.6x)
> 50GB 1 70.4 -> 69.9 (1x) 69.9 -> 47.1 (1.5x) 69.5 ->
> 41.8 (1.7x) 69.9 -> 36.6 (1.9x)
> 50GB 2 70.4 -> 70.2 (1x) 70.8 -> 47.8 (1.5x) 71.0 ->
> 42.9 (1.7x) 70.6 -> 40.6 (1.7x)
> 50GB 4 70.3 -> 70.5 (1x) 71.0 -> 47.6 (1.5x) 71.1 ->
> 42.9 (1.7x) 71.1 -> 40.2 (1.8x)
> ```
>
> ```
> VACUUM time, HEAD -> patched (gain)
> table indexes 0 workers 2 workers 4 workers
> 8 workers
> 10GB 0 12.1 -> 12.2 (1x) 12.1 -> 5.0 (2.4x) 12.1 ->
> 3.7 (3.3x) 12.2 -> 2.8 (4.3x)
> 10GB 1 26.0 -> 25.9 (1x) 25.6 -> 21.0 (1.2x) 25.6 ->
> 19.6 (1.3x) 25.8 -> 19.1 (1.3x)
> 10GB 2 33.0 -> 33.1 (1x) 28.3 -> 21.3 (1.3x) 28.3 ->
> 20.0 (1.4x) 28.2 -> 19.2 (1.5x)
> 10GB 4 48.0 -> 48.8 (1x) 38.1 -> 31.1 (1.2x) 28.9 ->
> 20.7 (1.4x) 29.0 -> 20.2 (1.4x)
> 50GB 0 60.3 -> 60.7 (1x) 60.3 -> 23.6 (2.6x) 60.5 ->
> 17.8 (3.4x) 60.8 -> 17.1 (3.6x)
> 50GB 1 134.3 -> 134.6 (1x) 133.3 -> 122.1 (1.1x) 133.4 ->
> 117.0 (1.1x) 133.1 -> 112.0 (1.2x)
> 50GB 2 190.8 -> 188.9 (1x) 161.8 -> 140.1 (1.2x) 161.4 ->
> 134.9 (1.2x) 161.5 -> 133.1 (1.2x)
> 50GB 4 326.4 -> 326.0 (1x) 276.4 -> 253.1 (1.1x) 236.8 ->
> 209.2 (1.1x) 236.6 -> 206.3 (1.1x)
> ```
>
> --
> Bharath Rupireddy
> Amazon Web Services: https://aws.amazon.com
>