Re: [WIP] Pipelined Recovery

(resending the mail, previous mail was held for moderation for some reason.)

Hi

I am attaching a new rebased version of the patch. Following are some
major changes in the new patch set.

* Streaming replication is now working. The prefetcher was not fully
decoupled from the startup process; that's why there were inconsistencies
in some scenarios and most of the recovery tap tests were failing.

* Patch is now split into consumer and producer patches. This will make
review easier.

* Pipeline shutdown flow is also improved. Now producer will always check
for the shutdown flag (being set by the consumer)

* Pipeline msg queue size is now configurable `wal_pipeline_queue_size`

* Tap tests now passes with PG_TEST_INITDB_EXTRA_OPTS="-c wal_pipeline=on"

* New tap test for `recovery/t/053_walpipeline.pl`. This covers some basic
functionality of the pipeline.

* The filename is changed to xlogpipeline.{h|C}

Thanks to all for the valuable feedback.
Looking forward to your reviews, comments, etc.

--
Regards,
Imran Zaheer

On Wed, Mar 18, 2026 at 3:15 PM Imran Zaheer <imran(dot)zhir(at)gmail(dot)com> wrote:

> (resending the mail, previous mail was held for moderation for some
> reason. Now pdf is moved to the tar.gz)
>
> Hi
>
> I am attaching a new rebased version of the patch. Following are some
> major changes in the new patch set.
>
> * Streaming replication is now working. The prefetcher was not fully
> decoupled from the startup process; that's why there were inconsistencies
> in some scenarios and most of the recovery tap tests were failing.
>
> * Patch is now split into consumer and producer patches. This will make
> review easier.
>
> * Pipeline shutdown flow is also improved. Now producer will always check
> for the shutdown flag (being set by the consumer)
>
> * Pipeline msg queue size is now configurable `wal_pipeline_queue_size`
>
> * Tap tests now passes with PG_TEST_INITDB_EXTRA_OPTS="-c wal_pipeline=on"
>
> * New tap test for `recovery/t/053_walpipeline.pl`. This covers some
> basic functionality of the pipeline.
>
> * The filename is changed to xlogpipeline.{h|C}
>
> Thanks to all for the valuable feedback.
> Looking forward to your reviews, comments, etc.
>
> --
> Regards,
> Imran Zaheer
>
>
> On Wed, Mar 18, 2026 at 12:43 PM Imran Zaheer <imran(dot)zhir(at)gmail(dot)com>
> wrote:
>
>> Hi Zsolt.
>>
>> Thanks alot for the review and pointing out the bugs. I have fixed the
>> bugs you mentioned in my new patch set. But
>> patchset mail is held for moderation for some reason.
>>
>> >
>> > if (reachedRecoveryTarget)
>> > {
>> > + if (wal_pipeline_enabled)
>> > + WalPipeline_Stop();
>> >
>> > What if we didn't reach the recovery target, shouldn't we stop the
>> > pipelines then?
>> >
>>
>> I have fixed the bugs shutdown logic.
>>
>> As we already know we will exist the recovery redo loop in
>> `PerformWalRecovery()` only in two cases
>>
>> 1: Recovery target reached:
>> In this case consumers will call to stop the pipeline.
>>
>> @@ -1807,6 +1931,9 @@ PerformWalRecovery(void)
>>
>> if (reachedRecoveryTarget)
>> {
>> + if (wal_pipeline_enabled)
>> + WalPipeline_Stop();
>> +
>>
>> 2: Available pg_wal is consumed and now more wal to read.
>> In this case pipeline producers will send the shutdown msg to the
>> consumer. Consumer will
>> detect this message and then call ` WalPipeline_Stop`. This is the case
>> where we cannot read
>> more records and the while loop will break here.
>>
>> + if (decoded_record)
>> + {
>> + record = &decoded_record->header;
>> + return record;
>> + }
>> + else
>> + {
>> + /*
>> + * We will end up here only when pipeline couldn't read more
>> + * records and have sent a shutdown msg. We will acknowldge this
>> + * and will trigger request to stop the pipeline workers.
>> + */
>> + WalPipeline_Stop();
>> + return NULL;
>> + }
>>
>>
>> Hope this makes sense.
>>
>> Once again thanks for reporting the bugs. You will receive the new
>> patchset mail soon once it is cleared from
>> the moderation.
>>
>> Looking forward to your reviews, comments, etc.
>>
>> Regards,
>> Imran Zaheer
>>
>