Re: [WIP] Pipelined Recovery

Hi

I am attaching a new series of patchset.

What's changed?

* Fixed two bugs that were causing failure for some tap tests with
wal_pipeline=on. First bug was that WalpipelineStop() was not calling
WakeupRecovery() that caused the producer to keep on waiting for the
new wal to arrive. This was causing pg_combinebackup/008_promote and
pg_basebackup/040_pg_createsubscriber to fail. Another bug in the
startup process was using stale values for `readFile` at the end of
recovery (FinishWalRecovery()) causing
recovery/035_standby_logical_decoding to fail.

* KnownAssignedTransactionIdsIdleMaintenance() is normally called by
the startup process before going into waiting. So far our pipeline we
were skipping KnownAssignedTransactionIdsIdleMaintenance() call
because pipeline worker was not able to call this func safely as
worker was not aware of the procarray related context (i.e. see
ProcArrayApplyRecoveryInfo()). This also caused
recovery/012_subtransactions to fail.

+ if (AmStartupProcess())
+ KnownAssignedTransactionIdsIdleMaintenance();

Now this is fixed by setting a flag before the producer starts the
wait in WaitForWALToBecomeAvailable(). The startup process detects
this flag and opurtunistically calls
KnownAssignedTransactionIdsIdleMaintenance() when all queue is
consumed. This way we will be able to call this from the consumer
(startup process) instead of the producer (decoder).

* New patch 0006, this will enable the walpipeline by default. This
way cfbot will be able to run the tests with pipelining on all
platforms. We can later decide whether to keep this feature on or off
by default. But for now this is gonna help cfbot find failures due to
the pipelining.

Other than that I forgot to mention about one change in my previous
mail. As earlier suggested by Jakub and Hayato, now the flamegraphs
attached in the benchmark zip will show reports based on per process
(pipeline bgw and startup process). This way the profiling will make
more sense instead of running perf on the whole cluster.

Little Summary of the idea:

Decoupling the decoder from the startup process and
offloading it to a dedicated worker. Then the startup process simply
receives the decoded records through a shmem queue. This allows the
startup process to spend most of the CPU time in applying the wal
records instead of decoding and reading them. We refer to this
approach as the wal pipelining.

Thanks
Imran Zaheer

Benchmarks: https://drive.google.com/file/d/13FATRT3kjh_y1wWETpYQh4ZXVLNaYU4A/view?usp=sharing
Test script: https://github.com/imranzaheer612/pg-recovery-testing