Re: Synchronizing slots from primary to standby

Hi,

On 11/13/23 5:24 AM, shveta malik wrote:
> On Thu, Nov 9, 2023 at 8:56 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
>>
>> Apart from the above, I would like to discuss the slot sync work
>> distribution strategy of this patch. The current implementation as
>> explained in the commit message [1] works well if the slots belong to
>> multiple databases. It is clear from the data in emails [2][3][4] that
>> having more workers really helps if the slots belong to multiple
>> databases. But I think if all the slots belong to one or very few
>> databases then such a strategy won't be as good. Now, on one hand, we
>> get very good numbers for a particular workload with the strategy used
>> in the patch but OTOH it may not be adaptable to various different
>> kinds of workloads. So, I have a question whether we should try to
>> optimize this strategy for various kinds of workloads or for the first
>> version let's use a single-slot sync-worker and then we can enhance
>> the functionality in later patches either in PG17 itself or in PG18 or
>> later versions.
>
> I can work on separating the patch. We can first focus on single
> worker design and then we can work on multi-worker design either
> immediately (if needed) or we can target it in the second draft of the
> patch. I would like to know the thoughts of others on this.

If we need to put more thoughts on the workers distribution strategy
then I also think it's better to focus on a single worker and then
improve/discuss a distribution design later on.

>
> One thing to note is that a lot of the complexity of
>> the patch is attributed to the multi-worker strategy which may still
>> not be efficient, so there is an argument to go with a simpler
>> single-slot sync-worker strategy and then enhance it in future
>> versions as we learn more about various workloads. It will also help
>> to develop this feature incrementally instead of doing all the things
>> in one go and taking a much longer time than it should.
>
> Agreed. With multi-workers, a lot of complexity (dsa, locks etc) have
> come into play. We can decide better on our workload distribution
> strategy among workers once we have more clarity on different types of
> workloads.
>

Agreed.

Regards,

--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com