Re: Proposal: Conflict log history table for Logical Replication

From: Dilip Kumar <dilipbalaut(at)gmail(dot)com>
To: Nisha Moond <nisha(dot)moond412(at)gmail(dot)com>
Cc: Amit Kapila <amit(dot)kapila16(at)gmail(dot)com>, shveta malik <shveta(dot)malik(at)gmail(dot)com>, vignesh C <vignesh21(at)gmail(dot)com>, Peter Smith <smithpb2250(at)gmail(dot)com>, Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com>, Bharath Rupireddy <bharath(dot)rupireddyforpostgres(at)gmail(dot)com>, PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org>
Subject: Re: Proposal: Conflict log history table for Logical Replication
Date: 2026-06-22 15:22:01
Message-ID: CAFiTN-tnBhCU8opWAjp1xYkjqGeXtQ1tfDZHbYBS7TifymXEvw@mail.gmail.com
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On Mon, Jun 22, 2026 at 4:32 PM Nisha Moond <nisha(dot)moond412(at)gmail(dot)com> wrote:
>
> On Sun, Jun 21, 2026 at 7:19 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> >
> > On Thu, Jun 18, 2026 at 9:33 AM shveta malik <shveta(dot)malik(at)gmail(dot)com> wrote:
> > >
> > > On Tue, Jun 16, 2026 at 6:54 PM Dilip Kumar <dilipbalaut(at)gmail(dot)com> wrote:
> > > >
> > > > IMHO we should just log WARNING and continue the apply worker on
> > > > conflict insertion failure, lets see what other thinks on this.
> > > >
> > >
> > > I have the same opinion. Allowing CLT to block the apply worker would
> > > be undesirable; CLT is a history/logs collection feature and should
> > > not interrupt core logical replication work.
> > >
> >
> > I think the insert can fail in rare cases like disk getting full while
> > writing WAL or some internal memory ERROR and the ERROR could be
> > persistent which means the LOG will be filled with the same WARNING if
> > there are many conflicts. Also, users may not like missing out on
> > conflict information. So, we can ERROR out and let users fix the
> > situation. Additionally, the nested try-catch to downgrade ERROR to
> > WARNING also looks ugly and a source of future bugs and maintenance
> > burden. The attached patch tries to fix this by ERRORing out on
> > insertion failure and attaching the required conflict info as a
> > context of ERROR. The patch also improved the ReportApplyConflict()
> > non-ERROR paths by displaying the conflict information in server LOGs
> > before inserting the same into CLT so that if insertion fails, the
> > complete conflict info can be present in server LOGs. See
> > v52-1-amit.Improve-error-handling-for-conflict-log-table-ins.
> >
> > Additionally, there is another problem with 0003 where when a parallel
> > apply worker hits an ERROR-level conflict it logs the conflict to the
> > conflict log table in a new transaction in its error path, after
> > aborting the failed apply transaction. But the leader detects worker
> > failure in pa_wait_for_xact_finish() by waiting on the worker's
> > transaction lock, and AbortOutOfAnyTransaction() releases that lock:
> > the leader unblocks, sees a non-finished state, raises "lost
> > connection to the logical replication parallel apply worker", and
> > tears the worker down -- which can SIGTERM it mid-insert and lose the
> > conflict log row, besides being a misleading message. The attached
> > top-up patch v52-2-amit.fix_parallel_apply_logging fixes that by
> > introducing PARALLEL_TRANS_ERROR state.
>
> I reproduced the above issue and verified the fix for it in
> v52-2-amit.fix_parallel_apply_logging. Here is a TAP test for the
> same.
> The attached top-up patch applies on top of the latest v53-0005 patch.
>
I have merged Amit's patch and Nisha's tap test patch and tested all
the cases discussed, and those are working fine.

--
Regards,
Dilip Kumar
Google

Attachment Content-Type Size
v54-0004-Preserve-conflict-log-destination-and-subscripti.patch application/octet-stream 23.7 KB
v54-0003-Implement-the-conflict-insertion-infrastructure-.patch application/octet-stream 45.0 KB
v54-0002-Report-error-for-ddls-on-conflict-log-tables.patch application/octet-stream 14.4 KB
v54-0005-Add-conflict-log-table-information-to-describe-s.patch application/octet-stream 78.7 KB
v54-0001-Add-configurable-conflict-log-table-for-Logical-.patch application/octet-stream 68.8 KB
v54-0006-Documentation-patch.patch application/octet-stream 42.3 KB

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Pavel Stehule 2026-06-22 15:30:59 Re: Global temporary tables
Previous Message Marcos Pegoraro 2026-06-22 14:54:38 Re: [PATCH] Add pg_get_table_ddl() to reconstruct CREATE TABLE statements