RE: Perform streaming logical transactions by background workers and parallel apply

From: "houzj(dot)fnst(at)fujitsu(dot)com" <houzj(dot)fnst(at)fujitsu(dot)com>
To: Amit Kapila <amit(dot)kapila16(at)gmail(dot)com>
Cc: Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com>, "wangw(dot)fnst(at)fujitsu(dot)com" <wangw(dot)fnst(at)fujitsu(dot)com>, Peter Smith <smithpb2250(at)gmail(dot)com>, "shiy(dot)fnst(at)fujitsu(dot)com" <shiy(dot)fnst(at)fujitsu(dot)com>, PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org>, Dilip Kumar <dilipbalaut(at)gmail(dot)com>
Subject: RE: Perform streaming logical transactions by background workers and parallel apply
Date: 2023-01-08 03:58:50
Message-ID: OS0PR01MB571685A645E509267AC5F30894F99@OS0PR01MB5716.jpnprd01.prod.outlook.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On Sunday, January 8, 2023 10:14 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
>
> On Sat, Jan 7, 2023 at 2:25 PM Dilip Kumar <dilipbalaut(at)gmail(dot)com> wrote:
> >
>
> Today, I was analyzing this patch w.r.t recent commit c6e1f62e2c and found that
> pa_set_xact_state() should set the latch (wake up) for the leader worker as the
> leader could be waiting in pa_wait_for_xact_state(). What do you think? But
> otherwise, it should be okay w.r.t DDLs because this patch allows the leader
> worker to restart logical replication for subscription parameter change which will
> in turn stop/restart parallel workers if required.

Thanks for the analysis. I agree that it would be better to signal the leader
when setting the state to PARALLEL_TRANS_STARTED, otherwise it might slightly
delay the timing of catch the state change in pa_wait_for_xact_state(), so I
have updated the patch for the same. Besides, I also checked commit c6e1f62e2c,
I think DDL operation doesn't need to wake up the parallel apply worker
directly as the parallel apply worker doesn't start table sync and only
communicate with the leader, so I didn't find some other places that need to be
changed.

Attach the updated patch set.

Best regards,
Hou zj

Attachment Content-Type Size
v77-0005-Add-a-main_worker_pid-to-pg_stat_subscription.patch application/octet-stream 9.4 KB
v77-0001-Perform-apply-of-large-transactions-by-parallel-.patch application/octet-stream 263.1 KB
v77-0002-Add-GUC-stream_serialize_threshold-and-test-seri.patch application/octet-stream 12.5 KB
v77-0003-Stop-extra-worker-if-GUC-was-changed.patch application/octet-stream 4.2 KB
v77-0004-Retry-to-apply-streaming-xact-only-in-apply-work.patch application/octet-stream 21.1 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Karl O. Pinc 2023-01-08 04:29:35 Re: drop postmaster symlink
Previous Message Andres Freund 2023-01-08 03:25:49 Re: pgsql: Delay commit status checks until freezing executes.