Re: [WIP] Pipelined Recovery

Hi

I am attaching a new series of the patch set.

What's changed?

* Rebased: patch v5-0004 is rebased. Just a small fix in meson.build

* Serialization:
Before this we were copying the decoded data and the msg header to a
temp buffer and passing it to the shmem mq via shm_mq_send(). The new
version uses shm_mq_sendv() with shm_mq_iovec instead. This avoids the
extra copy into the temporary buffer, as well as the additional
palloc()/pfree() required to allocate it. The changes would be
reflected in v5-0001 & a small portion (deserialization) of v5-0002.

* CPU resource usage:
New patch v5-0005 added to the patch set. Before this the resource
usage reported at the end of the recovery via pg_rusage_show() in
PerformWalRecovery() only reflected the startup process. Since WAL
decoding is now ofloaded to run on a separate pipeline worker, the CPU
time consumed by that worker was not included in the final report. The
new patch records the pipeline worker's CPU usage before it exits and
combines it with the startup process's CPU usage when the final
recovery statistics are reported. This ensures that the reported user
and system CPU times reflect the total CPU consumed during recovery,
while the elapsed time remains unchanged.

These are mostly refinement changes; the overall implementation and
design of the pipeline remain the same.

Summary of idea: Decoupling the decoding from the startup process and
offloading it to a dedicated worker. Then the startup process simply
receives the decoded record through a shmem queue. This allows the
startup process to spend most of the CPU time in applying the wal
records instead of decoding and reading them. We refer to this
approach as the wal pipeline.

Thanks
Imran Zaheer

Benchmarks: https://drive.google.com/file/d/13FATRT3kjh_y1wWETpYQh4ZXVLNaYU4A/view?usp=sharing
Test script: https://github.com/imranzaheer612/pg-recovery-testing

On Thu, Jun 25, 2026 at 12:47 PM Xuneng Zhou <xunengzhou(at)gmail(dot)com> wrote:
>
> Hi Imran,
>
> On Tue, Jun 23, 2026 at 9:27 PM Imran Zaheer <imran(dot)zhir(at)gmail(dot)com> wrote:
> >
> > Hi
> >
> > I am attaching the new series of patches.
> >
> > What has changed?
> >
> > * Rebased
> >
> > * The patch set is now split into two new patches. This will make the
> > code easier to understand and review.
> >
> > * The v4-0003 patch contains code mostly related to keeping the
> > recovery states synced between the startup process and the pipeline
> > process. Most of these changes were required to make the streaming
> > replication work.
> >
> > * The v4-0002 patch now only contains the consumer code that handles
> > receiving the decoded records from the shmem queue and moving the redo
> > loop forward.
> >
> > * The v4-0004 contains some basic tests to see if the pipeline worker
> > is functioning as expected. More testing was done by passing
> > PG_TEST_INITDB_EXTRA_OPTS="-c wal_pipeline=on" before running the
> > recovery test suite.
>
> +1 for splitting the patch set into smaller components to make the
> review process smoother.
>
> > * Other than that, the cpu overhead during deserialization is
> > optimized by skipping multiple copies of the decoded record and
> > directly passing the pointer to the shmem queue. There is still some
> > overhead visible during serialization that could be improved at the
> > producer end.
> >
> > * Signal handling for the pipeline worker is improved so that
> > promotion signals are sent to both the startup process and the
> > producer worker by the postmaster.
> >
> >
> > You will also find the new benchmarks attached [1] and the pdf report
> > overview. A simple cpu profiling on the pipelined startup process
> > shows that the cpu overhead during reading records has now been
> > removed and offloaded to the producer worker.
> >
> > Before pipelining:
> >
> > Around 50% of the cpu time is spent on fetching the wal record. Note that
> > in this workload pipeline is off so don't worry about the new func
> > ReceiveRecord(), it's just a wrapper around ReadRecord().
> >
> > Children Self Command Shared O Symbol
> > - 98.85% 0.21% postgres postgres [.] PerformWalRecovery
> > - 98.64% PerformWalRecovery
> > - 51.00% ReceiveRecord
> > - 50.78% ReadRecord
> > - 50.52% XLogPrefetcherReadRecord
> > - 49.61% XLogPrefetcherNextBlock
> > + 25.33% XLogReadAhead
> > + 22.32% PrefetchSharedBuffer
> > + 0.76% smgropen
> > - 46.68% ApplyWalRecord
> > + 29.23% heap_redo
> > + 9.51% heap2_redo
> > + 4.74% btree_redo
> > + 1.11% xlog_redo
> > + 0.80% xact_redo
> >
> >
> > After Pipelining:
> >
> > Here the only work needed to be done by the cpu is to get the decoded
> > record from
> > the queue. Other times (89.13%) cpu is worried about applying the wal record.
> >
> > Children Self Command Shared O Symbol
> > - 98.74% 0.37% postgres postgres [.] PerformWalRecovery
> > - 98.37% PerformWalRecovery
> > - 89.13% ApplyWalRecord
> > + 56.89% heap_redo
> > + 18.28% heap2_redo
> > + 8.01% btree_redo
> > + 2.02% xlog_redo
> > + 1.15% xact_redo
> > - 7.80% ReceiveRecord
> > + 7.63% WalPipeline_ReceiveRecord
> >
> > If the recovery process is not I/O bound then we would be able to test
> > this cpu optimization. Doing pgbench on a workload that is fully in
> > memory shows around 30% performance gains. You can see more
> > benchmarking details in the attached drive link [1]
>
> The perf result looks promising!
>
> > Some comments related to attached pdf and benchmarking, it is showing
> > that we can get more performance advantage out of the pipeline when
> > most of the workload is running in memory i.e. we have enough shared
> > buffers configured.
> >
> > If you want to do some experiments, please be my guest; I would be
> > happy to see more testing. You can share what performance advantage
> > you are getting from this. You can also refer to the benchmarking
> > script that I have been using [2].
> >
> >
> > Looking forward to your review, comments, etc.
>
> I haven't had a chance for a meaningful review yet, but expect to do so soon.
>
> --
> Regards,
> Xuneng Zhou
> HighGo Software Co., Ltd.