Re: Parallelize WAL Sender and WAL receiver

If speed create problem than make 3 easy script. 1. compress wal_archive.
2. ship wal archive over network and than 3. in the secondary server just
decompress it.

On Thu, Sep 5, 2024 at 9:18 AM Karthik Yellapragada <
karthik(dot)yellapragada(at)gmail(dot)com> wrote:

> Appreciate the responses, the network is good.. around 350 MB/s between
> prinary and secondary, both primary and secondary in the same DC. And it’s
> ASYC replication..
>
> I see the issue when the writes are super heavy ,
> I am only thinking if we can add more processes that sends / receives the
> WALs , we can speed up the transfer rate of WALs to the secondary..
>
> I understand the Apply can not be parallelized..
>
> On Wed, Sep 4, 2024 at 6:52 PM Rui DeSousa <rui(dot)desousa(at)icloud(dot)com> wrote:
>
>>
>>
>> > On Sep 4, 2024, at 9:03 PM, Karthik Yellapragada <
>> karthik(dot)yellapragada(at)gmail(dot)com> wrote:
>> >
>> > Hello all,
>> >
>> > I frequently face the problem of wals generated faster than the wals
>> transferred and applied.
>> >
>> > Is there a way to speed up the process? So I don’t accumulate a lot of
>> WALs at the primary?
>> >
>> > Regards,
>> > Karthik.
>>
>> What do you mean by parallelized WAL sender? Do you have many replicas
>> connecting to the primary? Have you tried cascading replication?
>>
>> What’s the network between the systems? Is a WAN with high latency? Can
>> the replica pull from the WAL file from an archive? For a high latency
>> networks like New York to London; I would terminate the WAL receiver via a
>> script if replication exceeded an acceptable delay to ensure SLA where
>> met. It is faster to pull the WAL files from the local archive which where
>> already replicated to London than streaming replication over a high latency
>> WAN. After It catches up, it will reconnect to streaming replication.
>>
>> --
> Thanks & Regards
> Karthik Yellapragada
> +1 860 830 5235
>