Re: enhancing pg_basebackup speeds up to ~23Gbps (small fixes + io_uring/Direct I/O)

On Tue, Oct 20, 2026 at 3:44 AM Gustavo William
<gustavowilliam0805(at)gmail(dot)com> wrote:
>
> Hi,
>
> Apologies for the delay.
>

Hi Gustavo, same from my side, apologies, finally I've got to this after
the PgConfEU. The 0005 didn't want to apply cleanly due to 51d7bf5edcaf4.
I've fixed it now.

> TL;DR:
>
> - The speedup from 0005 depends a lot on how much the incremental backup has changed
> since the last one: ~9.8x at 14% delta, ~3.7x at 33%, ~1.2x at 57%. That explains the
> difference between Jakub's 9.4x speedup and the ~1.5x I first saw.
> - Most of the gain comes from avoiding the kernel's own readahead on scattered blocks
> (it was doing more than double the I/O needed), not from concurrency.
> - Merging adjacent blocks matters. Without it, 0005 is slower than master on backups
> with a high delta.
> - maintenance_io_concurrency (m_io_c) beyond 2 only helps a little, and only when
> there are many adjacent blocks. m_io_c = 1 is still worse though.
> - Code changes in this version: prefetch is enabled again for m_io_c = 1, the merge
> limit is now io_combine_limit, there is a new limit on how far ahead we prefetch,
> and the prefetch state moved into a new IncrPrefetch struct.
>
> > I've verified ~9.4x improvement with 0005-incremental-backup
>
> I was confused by the big difference between our results, since I had initially seen
> only ~1.5x. After replicating your tests I found the reason: the speedup varies a lot
> depending on how much delta your incremental backup has.
>
> I followed the same method you described:
[..]
>
> delta change without 0005 with 0005 speedup
> ------------- ------------ --------- -------
> ~14% (~1.1G) 42.48 4.31 ~9.8x
> ~33% (~2.5G) 27.56 7.49 ~3.7x
> ~57% (~4.3G) 15.60 13.13 ~1.2x
>
> So the higher the delta, the smaller the gain from 0005. I think this ties back to the
> kernel behavior you observed.

True, I suspect that on production dbs, the delta is going to be rather small
most of the time.

> Reading scattered blocks the way pg_basebackup does today seems to confuse the kernel.
> After a single block read, the kernel may readahead the following blocks, expecting a
> sequential pattern. With scattered delta blocks that never happens, so the prefetched
> data is wasted. I actually checked this on /proc/<pid>/io during a backup of ~14% delta:
>
> Without 0005
> ```
> rchar: 1139995913 # bytes requested by the process (1.1GB)
> syscr: 138285
> read_bytes: 2557988864 # bytes actually read from storage (2.5GB)
> ```
>
> With 0005
> ```
> rchar: 1139966637 # bytes requested by the process (1.1GB)
> syscr: 138279
> read_bytes: 1143574528 # bytes actually read from storage (1.1GB)
> ```
>
> So, without 0005 the kernel does more than double the I/O we need. 0005 helps with
> concurrency, but its major improvement is avoiding this misguided readahead.

Thanks for doing those measurements - apparently we did not concentrate that
much on performance as we should, back when incrementental backups were
introduced (PG17).

> This explains the results above. With 14% delta from pgbench, most blocks are
> scattered, so 0005 avoids all that double I/O and gets a huge speedup. At 57% the delta
> blocks are much more adjacent, so the kernel's own readahead gets a good part of its
> guesses right and 0005 has less to improve on.
>
> A few other questions I tried to answer in this round:
>
> 1. Does increasing maintenance_io_concurrency make a big difference?
>
> Not really. There's some gain up to a point, but it's small. This also reinforces that
> most of the gain of 0005 comes from avoiding the kernel's double I/O, not from
> concurrency itself. Below are some results of backup with 14% delta (in seconds):
>
> with m_io_c = 2: 4.45
> with m_io_c = 16: 4.14
> with m_io_c = 32: 4.07
> with m_io_c = 64: 4.18
>
> This made me wonder if we should stop making the prefetch window tied to m_io_c and
> instead just use a small fixed number like 2. But a higher m_io_c does give a
> modest gain on backups with a higher delta (more adjacent blocks). With ~57% delta:
>
> with m_io_c = 2: 13.13
> with m_io_c = 16: 10.28
>

Let's just tie with m_io_c for simplicity. If it does not hurt, so we
can leave as that.

> 3. Is merging adjacent blocks really helping?
>
> At some point I considered dropping the merge-of-adjacent-blocks piece, so I decided
> to measure it on a 57% delta backup with m_io_c = 16 (results in seconds):
>
> without 0005: 15.60
> 0005 with merge: 10.28
> 0005 without merge: 20.41
>
> It helps a lot. Without it, 0005 makes incremental backups with a high delta
> slower than not having 0005 at all (15.60 above).
>

Cool.

>
> Now regarding the code changes of this last iteration:
>
> 2. We had previously agreed on a limit for how many blocks can be merged into a single
> fadvise. I was using maintenance_io_concurrency for that, but I thought
> io_combine_limit might be a better fit according to its description. Do you think
> that makes sense?

Yes, sounds good.

> 3. The previous version had no limit on how far ahead, in blocks, we could prefetch.
> Each iteration runs one prefetch_next_incremental_run(), which can cover many
> blocks when they're adjacent. With lots of adjacent blocks we could get far ahead of
> the block being read, filling the page cache with pages that won't be needed any
> time soon and might even be evicted before they are read.
>
> Based on that, I added a distance limit, in blocks, of
> maintenance_io_concurrency * io_combine_limit. When there are many adjacent blocks,
> that keeps us at most maintenance_io_concurrency fadvise calls ahead of the current
> block.

Sounds good too (unless anybody else complains of course). I've checked
syscalls using strace for fadvise64 vs pread64 and we seem to be good from
fadvise64 case. We could do way better for pread() for the incremental case
-- to merge adjacent indvidiual 8kB pread request into bigger ones/multuple
ones by using preadV(), but that's way of outside this fadvise-patch, and
it possibly would be smarter to use readstreaming API for that in far future
(I haven't investigated that).

> 4. I added a new struct named IncrPrefetch to the code to hold some parameters of the
> prefetch mechanism. I thought sendFile() was getting a lot of new variables and
> becoming harder to read. I hope that makes sense.

Sure!

Due to number of changes I'm attaching full current patchset changes include:
- your's new v0005 incremental (now 0006 temporarily) that was fixed due to
that 51d7bf5edcaf4
- fixes to re-reading & handling interruptions due to larger SINK_BUFF_SIZE
(0004, 0005) from paralllel subthread (as requested by Heikki)
- WIP-refactor of the sendFile and sendRelfile(); this prefetching patch
wouldn't apply later on, so I've rebased it further so now SINK_BUFF_SIZE
and related patches by Heikki are prerequsite for our fadvise patch)

-J.