Re: Proposal: Conflict log history table for Logical Replication

On Wed, Aug 26, 2026 at 1:52 PM vignesh C <vignesh21(at)gmail(dot)com> wrote:
>
> On Mon, 24 Aug 2026 at 21:01, Dilip Kumar <dilipbalaut(at)gmail(dot)com> wrote:
> >
> > On Mon, Aug 24, 2026 at 3:02 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> > >
> > > On Fri, Aug 21, 2026 at 12:13 PM vignesh C <vignesh21(at)gmail(dot)com> wrote:
> > > >
> > > > Thanks for the patch Dilip.
> > > > Logical replication breaks when conflict_log_destination = 'table' and
> > > > an update_origin_differs conflict involves a large row.
> > > > When the conflict occurs on a row containing a large value (~190 MB),
> > > > the conflict log table path converts the old row data into a string
> > > > representation. During this conversion, the resulting string buffer
> > > > grows beyond 1 GB and exceeds PostgreSQL's maximum StringInfo buffer
> > > > size of 1073741823 bytes.
> > > >
> > > > This results in:
> > > > "ERROR: string buffer exceeds maximum allowed length (1073741823 bytes)"
> > > > "DETAIL: Cannot enlarge string buffer containing 1073741808 bytes by
> > > > 32 more bytes."
> > > >
> > > > As a result, the logical replication apply worker exits, and
> > > > replication cannot make further progress.
> > > > Interestingly, the same scenario works successfully with
> > > > conflict_log_destination = 'log'. In that case, the conflict is logged
> > > > successfully and logical replication continues without any error.
> > > >
> > > > Reproducer
> > > > The issue can be reproduced with the following steps:
> > > > 1) Set up logical replication with conflict_log_destination = 'table'
> > > > and create the following table:
> > > > "CREATE TABLE tab (a int PRIMARY KEY, b text);"
> > > > 2) On the subscriber, enable track_commit_timestamp = on to detect
> > > > update_origin_differs conflicts.
> > > > 3) Insert a row on the publisher:
> > > > "INSERT INTO tab VALUES (1, 'small');"
> > > > 4) On the subscriber, update the replicated row with a large value:
> > > > "UPDATE tab SET b = repeat(chr(1), 190_000_000) WHERE a = 1;"
> > > >
> > >
> > > Here, the chr(1)'s special value makes the string longer. So, the
> > > failure is not really about row size, it is about JSON escaping
> > > expansion. escape_json* turns every byte below 0x20 into a six-byte
> > > \uXXXX sequence. The reproducer's value is 190MB of chr(1), which
> > > expands to ~1.14GB against the 1073741823-byte StringInfo limit,
> > > matching the reported error exactly. As per my understanding, this
> > > cannot be fixed by enlarging a buffer. The result has to become a json
> > > Datum, and varlena is capped at the same 1GB, so a tuple whose JSON
> > > form exceeds 1GB cannot be stored in a json column at all. It is a
> > > representational ceiling. So, the possible options to fix this are:
> > > (a) Don't LOG any column_value_size greater than 2K. (b) Truncate
> > > bigger column values(>2K) to 2K or even shorter number 64 bytes. (c)
> > > LOG only index/RI columns but what should we do for large columns on
> > > the subscriber side or if RI is set to FULL?
> > >
> > > If we choose option (b), then a truncated value in a queryable table
> > > silently produces wrong results: WHERE tuple->>'col' = ..., joins and
> > > old-vs-new diffs all answer incorrectly with no way for the consumer
> > > to tell. In a server log a human can see the value is clipped; in SQL
> > > nothing can. I think option (c) is not workable because of reasons
> > > mentioned along with option (c). I would prefer to go with option (a)
> > > aka store a value verbatim when it is below a cap; above the cap,
> > > record no value plus an explicit marker. Every value present in the
> > > CLT is then byte-exact, and every value that could not be stored says
> > > so.
> > >
> > > If we would like to go with option (a) then the next thing to decide
> > > is how to display/show/represent omitted columns.
> > >
> > > (1) The first option is to mark the omission on the value itself, e.g.
> > > "b": {"omitted": true, "length": 190000000}. This is self-locating
> > > (the same column can be oversized in one payload and fine in another),
> > > it cannot collide with a user column name (any scheme that adds a
> > > metadata key alongside column-name keys is unsafe, since a real column
> > > can be named anything), it travels with the row, and it fails loudly
> > > rather than looking like a genuine NULL. (2) The other option is to
> > > use a separate column like omitted_columns. There are three
> > > independent JSON payloads per row — replica_identity, remote_tuple,
> > > and local_conflicts (an array whose elements each contain a tuple).
> > > That's what creates the problems because this option can't say which
> > > payload. (3) Yet another option is to simply LOG the omitted column
> > > names but may be difficult for users to find out. Or maybe some
> > > combination of these options.
> > >
> > > The one benefit of option (2) is cheap filtering. WHERE
> > > omitted_columns IS NOT NULL to find incomplete rows is trivial,
> > > whereas with in-value markers you'd need a JSON search. So, I suggest
> > > to go with a mix of option 1 and 2, put the detail on the value — "b":
> > > {"omitted": true, "length": 190000000} and add a plain bool (say
> > > has_omitted_values) purely as a filter.
> >
> > Yes this makes sense. Here is the new version of patches which usage
> > v71_1-0001 as base patch and v71-0002 is rebased on top of that and
> > the the ommitting the large column patch is implemented as a top up
> > patch.
>
> Thanks Dilip for the updated patch, one suggestion:
> In update_missing and delete_missing, there are no local conflicting
> tuples. However, the current implementation adds an object with NULL
> values for every field:
> + * Builds the local conflicts JSON array column from the list of
> + * ConflictTupleInfo objects.
> + *
> + * Example output structure:
> + * [ { "xid": "1001", "commit_ts": "...", "origin": "...", "tuple":
> {...} }, ... ]
> + */
> +static Datum
> +build_local_conflicts_json_array(List *conflicttuples)
> +{
> + Datum *json_datum_array;
> + Datum json_array_datum;
> + int num_conflicts;
> + int i = 0;
> + int16 typlen;
> + bool typbyval;
> + char typalign;
> + TupleDesc tupdesc;
>
> - index_value = BuildIndexValueDescription(indexDesc, values, isnull);
> + /* Build local conflicts tuple descriptor. */
> + tupdesc = build_local_conflicts_tupledesc();
>
> This result in values such as:
> local_conflicts = [{"xid":null, "commit_ts":null, "origin":null, "tuple":null}]
>
> Would it be better to set local_conflicts itself to NULL in these
> cases? This would more clearly represent that there are no local
> conflicting tuples and allow users to simply check local_conflicts IS
> NOT NULL to identify conflicts involving a local tuple, rather than
> checking the individual fields.
>
> For example, the current output makes both cases appear as non-NULL
> local_conflicts:
> update_origin_differs | {"xid":"736", ... "tuple":{...}}
> delete_missing | {"xid":null, "commit_ts":null, "origin":null,
> "tuple":null}
> update_missing | {"xid":null, "commit_ts":null, "origin":null,
> "tuple":null}
>
> Using NULL for local_conflicts in the latter two cases would make the
> column's semantics clearer and the output easier for users to query.

The updated patch implements this suggestion and also fixes the issues
Shveta raised. And changed CONFLICT_MAX_VALUE_SIZE to 16KB and added
the comments, With CONFLICT_MAX_VALUE_SIZE set to 16kB, even a table
with the maximum supported number of columns (MaxHeapAttributeNumber =
1600) undergoing the worst-case 6x JSON escape expansion will produce
a serialized JSON datum of at most ~150MB, safely below PostgreSQL's
1GB varlena limit. Furthermore, across all three JSON columns in
ConflictLogSchema (replica_identity, remote_tuple, local_conflicts),
the total in-memory formed row size is bounded by ~460MB, ensuring it
stays comfortably within PostgreSQL's 1GB (MaxAllocSize) in-memory
row/tuple size limit.

I haven't yet addressed the latest issue Vignesh raised regarding
replica identity full, as Shveta already asked how an update could
succeed on the publisher without setting RI.

--
Regards,
Dilip Kumar
Google