Re: Parallel heap vacuum

Hi,

On Fri, Jul 17, 2026 at 10:03 AM Bharath Rupireddy
<bharath(dot)rupireddyforpostgres(at)gmail(dot)com> wrote:
>
> The main changes in this version:
>
> 1/ Moved the parallel-vacuum callbacks out of the table AM interface.
> As suggested upthread, the access method now passes them to
> vacuumparallel.c through a local callback struct, so the generic
> parallel vacuum code no longer calls into the table AM.

I have been working on this feature for a while now and discussing the
design with Sawada-san off-list.

A problem with the v20 patch's callback approach is that it requires
table AM implementers wanting to reuse the code to register their
parallel callbacks, which means the table AM shared library preload
and a restart. This doesn't sound like good interfacing.

So, keeping the design goal of reusing the parallel code from
vacuumparallel.c as much as possible, while keeping the layering
intact (vacuumlazy.c calling into vacuumparallel.c and not
vice-versa), I came up with the attached v21 patches. The idea is to
implement the parallel heap scan in the heap AM layer, i.e. in
vacuumlazy.c itself.

Please have a look at the attached 3 patches. 0001 moving the reusable
code to separate functions and structures, 0002 moving the common heap
scan state out of LVRelState, and 0003 implementing the parallel heap
scan (i.e. phase 1) for vacuum.

I benchmarked it (attached script) on an i4i.4xlarge EC2 instance with
local NVMe storage with a best case [1] to understand the
characteristics and contention points better. The patch makes the heap
scan 2.4x to 4.3x faster on the 10GB table (CPU bound) and 1.5x to
3.6x faster on the 50GB one (storage bound) with 2 to 8 workers, which
makes vacuum 1.1x to 1.5x faster on a table with indexes and up to
4.3x faster on one without.

I added some design notes in 0003's commit message and listed the ones
for some thoughts below:

1/ When the dataset fits in memory (s_b and RAM), the heap scan is CPU
bound and gains more. When it doesn't, it becomes storage bound and
gains less. With the default 2MB ring buffer, vacuum has to wait for
its own dirty buffers to be written out.

2/ Chunk size, the number of heap blocks each worker claims in one go,
causes VM page contention. A chunk size of 32K blocks, what a single
VM page covers, gains more, while 1K gives only up to 2.3x at 8
workers on the 10GB table when CPU bound. Since I used a chunk as the
eager scan region and as the point where a process checks the dead TID
store memory limit, I kept it fixed, so both stay the same size
through the scan.

3/ Phase 2 uses parallel workers and phase 3 stays serial, so the gain
in vacuum time is smaller, since phase 1 is only 25% to 50% of the
whole. With 4 indexes and 8 workers on the 10GB table, phase 1 is 43%
of the VACUUM time, so a 4x heap scan gain shows up as 1.4x overall.

4/ The cost of shared dead TID store lookups vs. a local store can be
reduced by caching the last block entry looked up in
TidStoreIsMember(). I will discuss it in a separate thread.

5/ The cost of the single exclusive lock per heap page on the whole
radix tree for the shared dead TID store is 11% of phase 1 at 4
workers and 17% at 8 workers on the 10GB table. I don't think we
benefit much from introducing radix tree optimizations for concurrent
writers such as OLC or ROWEX. I read the ART papers, and even if we
were to implement one, OLC seems to be enough to reduce write lock
contention, since vacuum doesn't have concurrent writers and readers
of the radix tree at the same time (phase 1 all writers, phases 2 and
3 all readers). ROWEX helps to minimize reader restarts when writers
are modifying concurrently, which may not be applicable as far as
vacuum is concerned.

Appreciate thoughts. Thanks.

[1] Test setup: i4i.4xlarge, 16 vCPU, 128GB RAM, local NVMe storage.
Unlogged table, 1 int column, 226 rows per page, 20% of rows deleted.
Each btree index is 6.2GB at 10GB and 31GB at 50GB. s_b 32GB, m_w_m
1GB, ring buffer 2MB, 32768-block chunk. Avg of 3 runs.

```
Phase 1, heap scan, HEAD -> patched (gain)
table indexes 0 workers 2 workers 4 workers
8 workers
10GB 0 12.1 -> 12.2 (1x) 12.1 -> 5.0 (2.4x) 12.1 ->
3.7 (3.3x) 12.2 -> 2.8 (4.3x)
10GB 1 12.2 -> 12.1 (1x) 12.0 -> 5.1 (2.4x) 12.0 ->
3.7 (3.2x) 12.2 -> 3.0 (4.0x)
10GB 2 12.0 -> 12.0 (1x) 12.2 -> 5.1 (2.4x) 12.2 ->
3.7 (3.3x) 12.2 -> 3.0 (4.0x)
10GB 4 12.0 -> 12.2 (1x) 12.3 -> 5.1 (2.4x) 12.3 ->
3.7 (3.3x) 12.3 -> 3.1 (4.0x)
50GB 0 60.3 -> 60.7 (1x) 60.3 -> 23.6 (2.6x) 60.5 ->
17.8 (3.4x) 60.8 -> 17.1 (3.6x)
50GB 1 70.4 -> 69.9 (1x) 69.9 -> 47.1 (1.5x) 69.5 ->
41.8 (1.7x) 69.9 -> 36.6 (1.9x)
50GB 2 70.4 -> 70.2 (1x) 70.8 -> 47.8 (1.5x) 71.0 ->
42.9 (1.7x) 70.6 -> 40.6 (1.7x)
50GB 4 70.3 -> 70.5 (1x) 71.0 -> 47.6 (1.5x) 71.1 ->
42.9 (1.7x) 71.1 -> 40.2 (1.8x)
```

```
VACUUM time, HEAD -> patched (gain)
table indexes 0 workers 2 workers 4 workers
8 workers
10GB 0 12.1 -> 12.2 (1x) 12.1 -> 5.0 (2.4x) 12.1 ->
3.7 (3.3x) 12.2 -> 2.8 (4.3x)
10GB 1 26.0 -> 25.9 (1x) 25.6 -> 21.0 (1.2x) 25.6 ->
19.6 (1.3x) 25.8 -> 19.1 (1.3x)
10GB 2 33.0 -> 33.1 (1x) 28.3 -> 21.3 (1.3x) 28.3 ->
20.0 (1.4x) 28.2 -> 19.2 (1.5x)
10GB 4 48.0 -> 48.8 (1x) 38.1 -> 31.1 (1.2x) 28.9 ->
20.7 (1.4x) 29.0 -> 20.2 (1.4x)
50GB 0 60.3 -> 60.7 (1x) 60.3 -> 23.6 (2.6x) 60.5 ->
17.8 (3.4x) 60.8 -> 17.1 (3.6x)
50GB 1 134.3 -> 134.6 (1x) 133.3 -> 122.1 (1.1x) 133.4 ->
117.0 (1.1x) 133.1 -> 112.0 (1.2x)
50GB 2 190.8 -> 188.9 (1x) 161.8 -> 140.1 (1.2x) 161.4 ->
134.9 (1.2x) 161.5 -> 133.1 (1.2x)
50GB 4 326.4 -> 326.0 (1x) 276.4 -> 253.1 (1.1x) 236.8 ->
209.2 (1.1x) 236.6 -> 206.3 (1.1x)
```

--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com