Re: Proposal: Conflict log history table for Logical Replication

On Mon, Aug 24, 2026 at 3:02 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
>
> On Fri, Aug 21, 2026 at 12:13 PM vignesh C <vignesh21(at)gmail(dot)com> wrote:
> >
> > Thanks for the patch Dilip.
> > Logical replication breaks when conflict_log_destination = 'table' and
> > an update_origin_differs conflict involves a large row.
> > When the conflict occurs on a row containing a large value (~190 MB),
> > the conflict log table path converts the old row data into a string
> > representation. During this conversion, the resulting string buffer
> > grows beyond 1 GB and exceeds PostgreSQL's maximum StringInfo buffer
> > size of 1073741823 bytes.
> >
> > This results in:
> > "ERROR: string buffer exceeds maximum allowed length (1073741823 bytes)"
> > "DETAIL: Cannot enlarge string buffer containing 1073741808 bytes by
> > 32 more bytes."
> >
> > As a result, the logical replication apply worker exits, and
> > replication cannot make further progress.
> > Interestingly, the same scenario works successfully with
> > conflict_log_destination = 'log'. In that case, the conflict is logged
> > successfully and logical replication continues without any error.
> >
> > Reproducer
> > The issue can be reproduced with the following steps:
> > 1) Set up logical replication with conflict_log_destination = 'table'
> > and create the following table:
> > "CREATE TABLE tab (a int PRIMARY KEY, b text);"
> > 2) On the subscriber, enable track_commit_timestamp = on to detect
> > update_origin_differs conflicts.
> > 3) Insert a row on the publisher:
> > "INSERT INTO tab VALUES (1, 'small');"
> > 4) On the subscriber, update the replicated row with a large value:
> > "UPDATE tab SET b = repeat(chr(1), 190_000_000) WHERE a = 1;"
> >
>
> Here, the chr(1)'s special value makes the string longer. So, the
> failure is not really about row size, it is about JSON escaping
> expansion. escape_json* turns every byte below 0x20 into a six-byte
> \uXXXX sequence. The reproducer's value is 190MB of chr(1), which
> expands to ~1.14GB against the 1073741823-byte StringInfo limit,
> matching the reported error exactly. As per my understanding, this
> cannot be fixed by enlarging a buffer. The result has to become a json
> Datum, and varlena is capped at the same 1GB, so a tuple whose JSON
> form exceeds 1GB cannot be stored in a json column at all. It is a
> representational ceiling. So, the possible options to fix this are:
> (a) Don't LOG any column_value_size greater than 2K. (b) Truncate
> bigger column values(>2K) to 2K or even shorter number 64 bytes. (c)
> LOG only index/RI columns but what should we do for large columns on
> the subscriber side or if RI is set to FULL?
>
> If we choose option (b), then a truncated value in a queryable table
> silently produces wrong results: WHERE tuple->>'col' = ..., joins and
> old-vs-new diffs all answer incorrectly with no way for the consumer
> to tell. In a server log a human can see the value is clipped; in SQL
> nothing can. I think option (c) is not workable because of reasons
> mentioned along with option (c). I would prefer to go with option (a)
> aka store a value verbatim when it is below a cap; above the cap,
> record no value plus an explicit marker. Every value present in the
> CLT is then byte-exact, and every value that could not be stored says
> so.
>
> If we would like to go with option (a) then the next thing to decide
> is how to display/show/represent omitted columns.
>
> (1) The first option is to mark the omission on the value itself, e.g.
> "b": {"omitted": true, "length": 190000000}. This is self-locating
> (the same column can be oversized in one payload and fine in another),
> it cannot collide with a user column name (any scheme that adds a
> metadata key alongside column-name keys is unsafe, since a real column
> can be named anything), it travels with the row, and it fails loudly
> rather than looking like a genuine NULL. (2) The other option is to
> use a separate column like omitted_columns. There are three
> independent JSON payloads per row — replica_identity, remote_tuple,
> and local_conflicts (an array whose elements each contain a tuple).
> That's what creates the problems because this option can't say which
> payload. (3) Yet another option is to simply LOG the omitted column
> names but may be difficult for users to find out. Or maybe some
> combination of these options.
>
> The one benefit of option (2) is cheap filtering. WHERE
> omitted_columns IS NOT NULL to find incomplete rows is trivial,
> whereas with in-value markers you'd need a JSON search. So, I suggest
> to go with a mix of option 1 and 2, put the detail on the value — "b":
> {"omitted": true, "length": 190000000} and add a plain bool (say
> has_omitted_values) purely as a filter.

Yes this makes sense. Here is the new version of patches which usage
v71_1-0001 as base patch and v71-0002 is rebased on top of that and
the the ommitting the large column patch is implemented as a top up
patch, which is slightly modified version(comments changes) of what
Amit shared offlist. In this, we use a 64kB max size for each column
and if a column's value exceeds this size, it will be omitted, as
suggested in option a. With a max value size of 64kB, even a table
with the maximum
supported number of columns (MaxHeapAttributeNumber = 1600) undergoing
the worst-case 6x JSON escape expansion will produce a serialized JSON
datum of at most ~600MB, safely below PostgreSQL's 1GB varlena limit.
Furthermore, across all three JSON columns in ConflictLogSchema
(replica_identity, remote_tuple, local_conflicts), the total in-memory
formed row size is bounded by ~1.8GB, ensuring it stays within
PostgreSQL's 2GB row size limit (assuming local_conflicts contains at
most one local tuple, which holds for all resolvable conflicts today).

--
Regards,
Dilip Kumar
Google