Re: Proposal: Conflict log history table for Logical Replication

On Fri, Aug 21, 2026 at 12:13 PM vignesh C <vignesh21(at)gmail(dot)com> wrote:
>
> Thanks for the patch Dilip.
> Logical replication breaks when conflict_log_destination = 'table' and
> an update_origin_differs conflict involves a large row.
> When the conflict occurs on a row containing a large value (~190 MB),
> the conflict log table path converts the old row data into a string
> representation. During this conversion, the resulting string buffer
> grows beyond 1 GB and exceeds PostgreSQL's maximum StringInfo buffer
> size of 1073741823 bytes.
>
> This results in:
> "ERROR: string buffer exceeds maximum allowed length (1073741823 bytes)"
> "DETAIL: Cannot enlarge string buffer containing 1073741808 bytes by
> 32 more bytes."
>
> As a result, the logical replication apply worker exits, and
> replication cannot make further progress.
> Interestingly, the same scenario works successfully with
> conflict_log_destination = 'log'. In that case, the conflict is logged
> successfully and logical replication continues without any error.
>
> Reproducer
> The issue can be reproduced with the following steps:
> 1) Set up logical replication with conflict_log_destination = 'table'
> and create the following table:
> "CREATE TABLE tab (a int PRIMARY KEY, b text);"
> 2) On the subscriber, enable track_commit_timestamp = on to detect
> update_origin_differs conflicts.
> 3) Insert a row on the publisher:
> "INSERT INTO tab VALUES (1, 'small');"
> 4) On the subscriber, update the replicated row with a large value:
> "UPDATE tab SET b = repeat(chr(1), 190_000_000) WHERE a = 1;"
>

Here, the chr(1)'s special value makes the string longer. So, the
failure is not really about row size, it is about JSON escaping
expansion. escape_json* turns every byte below 0x20 into a six-byte
\uXXXX sequence. The reproducer's value is 190MB of chr(1), which
expands to ~1.14GB against the 1073741823-byte StringInfo limit,
matching the reported error exactly. As per my understanding, this
cannot be fixed by enlarging a buffer. The result has to become a json
Datum, and varlena is capped at the same 1GB, so a tuple whose JSON
form exceeds 1GB cannot be stored in a json column at all. It is a
representational ceiling. So, the possible options to fix this are:
(a) Don't LOG any column_value_size greater than 2K. (b) Truncate
bigger column values(>2K) to 2K or even shorter number 64 bytes. (c)
LOG only index/RI columns but what should we do for large columns on
the subscriber side or if RI is set to FULL?

If we choose option (b), then a truncated value in a queryable table
silently produces wrong results: WHERE tuple->>'col' = ..., joins and
old-vs-new diffs all answer incorrectly with no way for the consumer
to tell. In a server log a human can see the value is clipped; in SQL
nothing can. I think option (c) is not workable because of reasons
mentioned along with option (c). I would prefer to go with option (a)
aka store a value verbatim when it is below a cap; above the cap,
record no value plus an explicit marker. Every value present in the
CLT is then byte-exact, and every value that could not be stored says
so.

If we would like to go with option (a) then the next thing to decide
is how to display/show/represent omitted columns.

(1) The first option is to mark the omission on the value itself, e.g.
"b": {"omitted": true, "length": 190000000}. This is self-locating
(the same column can be oversized in one payload and fine in another),
it cannot collide with a user column name (any scheme that adds a
metadata key alongside column-name keys is unsafe, since a real column
can be named anything), it travels with the row, and it fails loudly
rather than looking like a genuine NULL. (2) The other option is to
use a separate column like omitted_columns. There are three
independent JSON payloads per row — replica_identity, remote_tuple,
and local_conflicts (an array whose elements each contain a tuple).
That's what creates the problems because this option can't say which
payload. (3) Yet another option is to simply LOG the omitted column
names but may be difficult for users to find out. Or maybe some
combination of these options.

The one benefit of option (2) is cheap filtering. WHERE
omitted_columns IS NOT NULL to find incomplete rows is trivial,
whereas with in-value markers you'd need a JSON search. So, I suggest
to go with a mix of option 1 and 2, put the detail on the value — "b":
{"omitted": true, "length": 190000000} and add a plain bool (say
has_omitted_values) purely as a filter.

--
With Regards,
Amit Kapila.