Re: Proposal: Conflict log history table for Logical Replication

On Thu, Sep 11, 2025 at 12:53 AM Bharath Rupireddy
<bharath(dot)rupireddyforpostgres(at)gmail(dot)com> wrote:
>
> On Tue, Aug 5, 2025 at 5:24 AM Dilip Kumar <dilipbalaut(at)gmail(dot)com> wrote:
> >
> > Currently we log conflicts to the server's log file and updates, this
> > approach has limitations, 1) Difficult to query and analyze, parsing
> > plain text log files for conflict details is inefficient. 2) Lack of
> > structured data, key conflict attributes (table, operation, old/new
> > data, LSN, etc.) are not readily available in a structured, queryable
> > format. 3) Difficult for external monitoring tools or custom
> > resolution scripts to consume conflict data directly.
> >
> > This proposal aims to address these limitations by introducing a
> > conflict log history table, providing a structured, and queryable
> > record of all logical replication conflicts. This should be a
> > configurable option whether to log into the conflict log history
> > table, server logs or both.
>
> +1 for the overall idea. Having an option to separate out the
> conflicts helps analyze the data correctness issues and understand the
> behavior of conflicts.
>
> Parsing server logs file for analysis and debugging is a typical
> requirement differently met with tools like log_fdw or capture server
> logs in CSV format for parsing or do text search and analyze etc.
>
> > This proposal has two main design questions:
> > ===================================
> >
> > 1. How do we store conflicting tuples from different tables?
> > Using a JSON column to store the row data seems like the most flexible
> > solution, as it can accommodate different table schemas.
>
> How good is storing conflicts on the table? Is it okay to generate WAL
> traffic?
>

Yesh, I think so. One would like to query conflicts and resolutions
for those conflicts at a later point to ensure consistency. BTW, if
you are worried about WAL traffic, please note conflicts shouldn't be
a very often event, so additional WAL should be okay. OTOH, if the
conflicts are frequent, anyway, the performance won't be that great as
that means there is a kind of ERROR which we have to deal by having
resolution for it.

> Is it okay to physically replicate this log table to all
> replicas?
>

Yes, that should be okay as we want the conflict_tables to be present
after failover.

Is it okay to logically replicate this log table to all
> subscribers and logical decoding clients?
>

I think we should avoid this.

> How does this table get
> truncated? If truncation gets delayed, won't it unnecessarily fill up
> storage?
>

I think it should be users responsibility to clean this table as they
better know when the data in the table is obsolete. Eventually, we can
also have some policies via options or some other way to get it
truncated. IIRC, we also discussed having these as partition tables so
that it is easy to discard data. However, for initial version, we may
want something simpler.

> > 2. Should this be a system table or a user table?
> > a) System Table: Storing this in a system catalog is simple, but
> > catalogs aren't designed for ever-growing data. While pg_large_object
> > is an exception, this is not what we generally do IMHO.
> > b) User Table: This offers more flexibility. We could allow a user to
> > specify the table name during CREATE SUBSCRIPTION. Then we choose to
> > either create the table internally or let the user create the table
> > with a predefined schema.
>
> -1 for the system table for sure.
>
> > A potential drawback is that a user might drop or alter the table.
> > However, we could mitigate this risk by simply logging a WARNING if
> > the table is configured but an insertion fails.
> > I am currently working on a POC patch for the same, but will post that
> > once we have some thoughts on design choices.
>
> How about streaming the conflicts in fixed format to a separate log
> file other than regular postgres server log file?
>

I would prefer this info to be stored in tables as it would be easy to
query them. If we use separate LOGs then we should provide some views
to query the LOG.

--
With Regards,
Amit Kapila.