Re: Support logical replication of DDLs

From: shveta malik <shveta(dot)malik(at)gmail(dot)com>
To: Amit Kapila <amit(dot)kapila16(at)gmail(dot)com>
Cc: "Yu Shi (Fujitsu)" <shiy(dot)fnst(at)fujitsu(dot)com>, vignesh C <vignesh21(at)gmail(dot)com>, "Zhijie Hou (Fujitsu)" <houzj(dot)fnst(at)fujitsu(dot)com>, Ajin Cherian <itsajin(at)gmail(dot)com>, "Wei Wang (Fujitsu)" <wangw(dot)fnst(at)fujitsu(dot)com>, Runqi Tian <runqidev(at)gmail(dot)com>, Peter Smith <smithpb2250(at)gmail(dot)com>, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, li jie <ggysxcq(at)gmail(dot)com>, Dilip Kumar <dilipbalaut(at)gmail(dot)com>, Alvaro Herrera <alvherre(at)alvh(dot)no-ip(dot)org>, Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com>, Japin Li <japinli(at)hotmail(dot)com>, rajesh singarapu <rajesh(dot)rs0541(at)gmail(dot)com>, PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org>, Zheng Li <zhengli10(at)gmail(dot)com>, shveta malik <shveta(dot)malik(at)gmail(dot)com>
Subject: Re: Support logical replication of DDLs
Date: 2023-06-08 12:01:49
Message-ID: CAJpy0uDtrxPo127LH3FP-TffynrspPFqhhC7o_GFOMP+2mPtWQ@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general pgsql-hackers

On Thu, Jun 8, 2023 at 10:36 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
>
> On Tue, Jun 6, 2023 at 4:26 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> >
> >
> > Few assorted comments:
> > ===================
> >
>
> Few comments on 0008* patch:
> ==============================
> 1. After 0008*, deparse_CreateStmt(), forms dpcontext before forming
> identity whereas it is required only after it. It may not matter
> practically but it is better to do the work when it is required. Also,
> before 0008, it appears to be formed after identity, so not sure if
> the change in 0008 is intentional, if so, then please let me know the
> reason, and may be it is better to add a comment for the same.
>

Shifted dpcontext creation after identity.

> 2. Similarly, what is need to separately do insert_identity_object()
> in deparse_CreateStmt() instead of directly doing something like
> new_objtree_for_qualname() as we are doing in 0001 patch? For this, I
> guess you need objtype handling in new_jsonb_VA().
>

yes, for that we need object handling in new_jsonb_VA(), which is not
possible to do in this case as there is no standard format for the
jsonb object. As in this case, it is identity object which is of
format "identity": {"objname": "test1", "schemaname": "public"}, while
in some other case object could be say coltype which is of format :
"coltype": {"typmod": "", "typarray": false, "typename": "int4",
"schemaname": "pg_catalog"}. And we need to push each element of
these objects to output jsonbParseState as and when we read the
parse-tree instead of saving them to an intermediate structure and
then reading from that. This makes us construct json-objects outside
of new_jsonb_VA().

> 3.
> /*
> * It will be of array type for multi-columns table, so lets begin
> * an arrayobject. deparse_TableElems_ToJsonb() will add elements
> * to it.
> */
> pushJsonbValue(&state, WJB_BEGIN_ARRAY, NULL);
>
> deparse_TableElems_ToJsonb(state, relation, node->tableElts, dpcontext,
> false, /* not typed table */
> false); /* not composite */
> deparse_Constraints_ToJsonb(state, objectId);
>
> pushJsonbValue(&state, WJB_END_ARRAY, NULL);
>
> This part of the code is repeated in deparse_CreateStmt(). Can we move
> this to a separate function?
>

Common code shifted to another function insert_table_elements().
On similar line. added another new function table_elem_present() to
encapsulate 2 calls for the column and constraint into it.

> 4.
> * Note we ignore constraints in the parse node here; they are extracted from
> * system catalogs instead.
> */
>
> static void
> deparse_TableElems_ToJsonb(JsonbParseState *state, Relation relation,
>
> An extra empty line between the comments end and function appears unnecessary.
>

Modified.

> 5.
> + /* creat name and type elements for column */
>
> /creat/create
>

Modified.

Please find new set of patches addressing below:
a) Issue mentioned by Wang-san in [1],
b) Comments from Peter given in [2]
c) Comments from Amit given in the last 2 emails.

[1]: https://www.postgresql.org/message-id/OS3PR01MB62750D43D4F7F075B33BD2609E52A%40OS3PR01MB6275.jpnprd01.prod.outlook.com
[2]: https://www.postgresql.org/message-id/CAHut%2BPv9vPbUQc0fzrKmDkKOsS_bj-hup_E%2BsLHNEX%2B6F%2BSY5Q%40mail.gmail.com

Thank You Vignesh for handling (a), Ajin for handling (b), Shi-san and
Hou-san for contributing in (c).

The new changes are in patch 0001, 0002, 0005 and 0008.

thanks
Shveta

Attachment Content-Type Size
0001-Deparser-for-Table-DDL-commands-and-exten-2023_06_08.patch application/octet-stream 128.7 KB
0002-Enhance-the-event-trigger-to-support-DDL--2023_06_08.patch application/octet-stream 63.7 KB
0005-DDL-replication-for-Table-DDL-commands-2023_06_08.patch application/octet-stream 215.8 KB
0003-Add-verbose-option-for-ddl-deparse-module-2023_06_08.patch application/octet-stream 47.1 KB
0004-Introduce-the-test_ddl_deparse_regress-te-2023_06_08.patch application/octet-stream 977.8 KB
0006-Add-subscription-tap-test-for-DDL-replica-2023_06_08.patch application/octet-stream 20.8 KB
0007-Apply-the-DDL-change-as-that-same-user-th-2023_06_08.patch application/octet-stream 59.9 KB
0008-ObjTree-Removal-for-multiple-commands-2023_06_08.patch application/octet-stream 1.8 MB

In response to

Responses

Browse pgsql-general by date

  From Date Subject
Next Message shveta malik 2023-06-08 12:07:58 Re: Support logical replication of DDLs
Previous Message Amit Kapila 2023-06-08 05:05:54 Re: Support logical replication of DDLs

Browse pgsql-hackers by date

  From Date Subject
Next Message Tomas Vondra 2023-06-08 12:03:47 Re: BRIN indexes vs. SK_SEARCHARRAY (and preprocessing scan keys)
Previous Message Jose Luis Tallon 2023-06-08 12:01:16 Re: Let's make PostgreSQL multi-threaded