Re: Proposal: Conflict log history table for Logical Replication

On Mon, 24 Aug 2026 at 21:01, Dilip Kumar <dilipbalaut(at)gmail(dot)com> wrote:
>
> On Mon, Aug 24, 2026 at 3:02 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> >
> > On Fri, Aug 21, 2026 at 12:13 PM vignesh C <vignesh21(at)gmail(dot)com> wrote:
> > >
> > > Thanks for the patch Dilip.
> > > Logical replication breaks when conflict_log_destination = 'table' and
> > > an update_origin_differs conflict involves a large row.
> > > When the conflict occurs on a row containing a large value (~190 MB),
> > > the conflict log table path converts the old row data into a string
> > > representation. During this conversion, the resulting string buffer
> > > grows beyond 1 GB and exceeds PostgreSQL's maximum StringInfo buffer
> > > size of 1073741823 bytes.
> > >
> > > This results in:
> > > "ERROR: string buffer exceeds maximum allowed length (1073741823 bytes)"
> > > "DETAIL: Cannot enlarge string buffer containing 1073741808 bytes by
> > > 32 more bytes."
> > >
> > > As a result, the logical replication apply worker exits, and
> > > replication cannot make further progress.
> > > Interestingly, the same scenario works successfully with
> > > conflict_log_destination = 'log'. In that case, the conflict is logged
> > > successfully and logical replication continues without any error.
> > >
> > > Reproducer
> > > The issue can be reproduced with the following steps:
> > > 1) Set up logical replication with conflict_log_destination = 'table'
> > > and create the following table:
> > > "CREATE TABLE tab (a int PRIMARY KEY, b text);"
> > > 2) On the subscriber, enable track_commit_timestamp = on to detect
> > > update_origin_differs conflicts.
> > > 3) Insert a row on the publisher:
> > > "INSERT INTO tab VALUES (1, 'small');"
> > > 4) On the subscriber, update the replicated row with a large value:
> > > "UPDATE tab SET b = repeat(chr(1), 190_000_000) WHERE a = 1;"
> > >
> >
> > Here, the chr(1)'s special value makes the string longer. So, the
> > failure is not really about row size, it is about JSON escaping
> > expansion. escape_json* turns every byte below 0x20 into a six-byte
> > \uXXXX sequence. The reproducer's value is 190MB of chr(1), which
> > expands to ~1.14GB against the 1073741823-byte StringInfo limit,
> > matching the reported error exactly. As per my understanding, this
> > cannot be fixed by enlarging a buffer. The result has to become a json
> > Datum, and varlena is capped at the same 1GB, so a tuple whose JSON
> > form exceeds 1GB cannot be stored in a json column at all. It is a
> > representational ceiling. So, the possible options to fix this are:
> > (a) Don't LOG any column_value_size greater than 2K. (b) Truncate
> > bigger column values(>2K) to 2K or even shorter number 64 bytes. (c)
> > LOG only index/RI columns but what should we do for large columns on
> > the subscriber side or if RI is set to FULL?
> >
> > If we choose option (b), then a truncated value in a queryable table
> > silently produces wrong results: WHERE tuple->>'col' = ..., joins and
> > old-vs-new diffs all answer incorrectly with no way for the consumer
> > to tell. In a server log a human can see the value is clipped; in SQL
> > nothing can. I think option (c) is not workable because of reasons
> > mentioned along with option (c). I would prefer to go with option (a)
> > aka store a value verbatim when it is below a cap; above the cap,
> > record no value plus an explicit marker. Every value present in the
> > CLT is then byte-exact, and every value that could not be stored says
> > so.
> >
> > If we would like to go with option (a) then the next thing to decide
> > is how to display/show/represent omitted columns.
> >
> > (1) The first option is to mark the omission on the value itself, e.g.
> > "b": {"omitted": true, "length": 190000000}. This is self-locating
> > (the same column can be oversized in one payload and fine in another),
> > it cannot collide with a user column name (any scheme that adds a
> > metadata key alongside column-name keys is unsafe, since a real column
> > can be named anything), it travels with the row, and it fails loudly
> > rather than looking like a genuine NULL. (2) The other option is to
> > use a separate column like omitted_columns. There are three
> > independent JSON payloads per row — replica_identity, remote_tuple,
> > and local_conflicts (an array whose elements each contain a tuple).
> > That's what creates the problems because this option can't say which
> > payload. (3) Yet another option is to simply LOG the omitted column
> > names but may be difficult for users to find out. Or maybe some
> > combination of these options.
> >
> > The one benefit of option (2) is cheap filtering. WHERE
> > omitted_columns IS NOT NULL to find incomplete rows is trivial,
> > whereas with in-value markers you'd need a JSON search. So, I suggest
> > to go with a mix of option 1 and 2, put the detail on the value — "b":
> > {"omitted": true, "length": 190000000} and add a plain bool (say
> > has_omitted_values) purely as a filter.
>
> Yes this makes sense. Here is the new version of patches which usage
> v71_1-0001 as base patch and v71-0002 is rebased on top of that and
> the the ommitting the large column patch is implemented as a top up
> patch.

Thanks Dilip for the updated patch, one suggestion:
In update_missing and delete_missing, there are no local conflicting
tuples. However, the current implementation adds an object with NULL
values for every field:
+ * Builds the local conflicts JSON array column from the list of
+ * ConflictTupleInfo objects.
+ *
+ * Example output structure:
+ * [ { "xid": "1001", "commit_ts": "...", "origin": "...", "tuple":
{...} }, ... ]
+ */
+static Datum
+build_local_conflicts_json_array(List *conflicttuples)
+{
+ Datum *json_datum_array;
+ Datum json_array_datum;
+ int num_conflicts;
+ int i = 0;
+ int16 typlen;
+ bool typbyval;
+ char typalign;
+ TupleDesc tupdesc;

- index_value = BuildIndexValueDescription(indexDesc, values, isnull);
+ /* Build local conflicts tuple descriptor. */
+ tupdesc = build_local_conflicts_tupledesc();

This result in values such as:
local_conflicts = [{"xid":null, "commit_ts":null, "origin":null, "tuple":null}]

Would it be better to set local_conflicts itself to NULL in these
cases? This would more clearly represent that there are no local
conflicting tuples and allow users to simply check local_conflicts IS
NOT NULL to identify conflicts involving a local tuple, rather than
checking the individual fields.

For example, the current output makes both cases appear as non-NULL
local_conflicts:
update_origin_differs | {"xid":"736", ... "tuple":{...}}
delete_missing | {"xid":null, "commit_ts":null, "origin":null,
"tuple":null}
update_missing | {"xid":null, "commit_ts":null, "origin":null,
"tuple":null}

Using NULL for local_conflicts in the latter two cases would make the
column's semantics clearer and the output easier for users to query.

Regards,
Vignesh