Re: Proposal: Conflict log history table for Logical Replication

On Mon, Aug 24, 2026 at 9:01 PM Dilip Kumar <dilipbalaut(at)gmail(dot)com> wrote:
>
> On Mon, Aug 24, 2026 at 3:02 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> >
> > On Fri, Aug 21, 2026 at 12:13 PM vignesh C <vignesh21(at)gmail(dot)com> wrote:
> > >
> > > Thanks for the patch Dilip.
> > > Logical replication breaks when conflict_log_destination = 'table' and
> > > an update_origin_differs conflict involves a large row.
> > > When the conflict occurs on a row containing a large value (~190 MB),
> > > the conflict log table path converts the old row data into a string
> > > representation. During this conversion, the resulting string buffer
> > > grows beyond 1 GB and exceeds PostgreSQL's maximum StringInfo buffer
> > > size of 1073741823 bytes.
> > >
> > > This results in:
> > > "ERROR: string buffer exceeds maximum allowed length (1073741823 bytes)"
> > > "DETAIL: Cannot enlarge string buffer containing 1073741808 bytes by
> > > 32 more bytes."
> > >
> > > As a result, the logical replication apply worker exits, and
> > > replication cannot make further progress.
> > > Interestingly, the same scenario works successfully with
> > > conflict_log_destination = 'log'. In that case, the conflict is logged
> > > successfully and logical replication continues without any error.
> > >
> > > Reproducer
> > > The issue can be reproduced with the following steps:
> > > 1) Set up logical replication with conflict_log_destination = 'table'
> > > and create the following table:
> > > "CREATE TABLE tab (a int PRIMARY KEY, b text);"
> > > 2) On the subscriber, enable track_commit_timestamp = on to detect
> > > update_origin_differs conflicts.
> > > 3) Insert a row on the publisher:
> > > "INSERT INTO tab VALUES (1, 'small');"
> > > 4) On the subscriber, update the replicated row with a large value:
> > > "UPDATE tab SET b = repeat(chr(1), 190_000_000) WHERE a = 1;"
> > >
> >
> > Here, the chr(1)'s special value makes the string longer. So, the
> > failure is not really about row size, it is about JSON escaping
> > expansion. escape_json* turns every byte below 0x20 into a six-byte
> > \uXXXX sequence. The reproducer's value is 190MB of chr(1), which
> > expands to ~1.14GB against the 1073741823-byte StringInfo limit,
> > matching the reported error exactly. As per my understanding, this
> > cannot be fixed by enlarging a buffer. The result has to become a json
> > Datum, and varlena is capped at the same 1GB, so a tuple whose JSON
> > form exceeds 1GB cannot be stored in a json column at all. It is a
> > representational ceiling. So, the possible options to fix this are:
> > (a) Don't LOG any column_value_size greater than 2K. (b) Truncate
> > bigger column values(>2K) to 2K or even shorter number 64 bytes. (c)
> > LOG only index/RI columns but what should we do for large columns on
> > the subscriber side or if RI is set to FULL?
> >
> > If we choose option (b), then a truncated value in a queryable table
> > silently produces wrong results: WHERE tuple->>'col' = ..., joins and
> > old-vs-new diffs all answer incorrectly with no way for the consumer
> > to tell. In a server log a human can see the value is clipped; in SQL
> > nothing can. I think option (c) is not workable because of reasons
> > mentioned along with option (c). I would prefer to go with option (a)
> > aka store a value verbatim when it is below a cap; above the cap,
> > record no value plus an explicit marker. Every value present in the
> > CLT is then byte-exact, and every value that could not be stored says
> > so.
> >
> > If we would like to go with option (a) then the next thing to decide
> > is how to display/show/represent omitted columns.
> >
> > (1) The first option is to mark the omission on the value itself, e.g.
> > "b": {"omitted": true, "length": 190000000}. This is self-locating
> > (the same column can be oversized in one payload and fine in another),
> > it cannot collide with a user column name (any scheme that adds a
> > metadata key alongside column-name keys is unsafe, since a real column
> > can be named anything), it travels with the row, and it fails loudly
> > rather than looking like a genuine NULL. (2) The other option is to
> > use a separate column like omitted_columns. There are three
> > independent JSON payloads per row — replica_identity, remote_tuple,
> > and local_conflicts (an array whose elements each contain a tuple).
> > That's what creates the problems because this option can't say which
> > payload. (3) Yet another option is to simply LOG the omitted column
> > names but may be difficult for users to find out. Or maybe some
> > combination of these options.
> >
> > The one benefit of option (2) is cheap filtering. WHERE
> > omitted_columns IS NOT NULL to find incomplete rows is trivial,
> > whereas with in-value markers you'd need a JSON search. So, I suggest
> > to go with a mix of option 1 and 2, put the detail on the value — "b":
> > {"omitted": true, "length": 190000000} and add a plain bool (say
> > has_omitted_values) purely as a filter.
>
> Yes this makes sense. Here is the new version of patches which usage
> v71_1-0001 as base patch and v71-0002 is rebased on top of that and
> the the ommitting the large column patch is implemented as a top up
> patch, which is slightly modified version(comments changes) of what
> Amit shared offlist. In this, we use a 64kB max size for each column
> and if a column's value exceeds this size, it will be omitted, as
> suggested in option a. With a max value size of 64kB, even a table
> with the maximum
> supported number of columns (MaxHeapAttributeNumber = 1600) undergoing
> the worst-case 6x JSON escape expansion will produce a serialized JSON
> datum of at most ~600MB, safely below PostgreSQL's 1GB varlena limit.
> Furthermore, across all three JSON columns in ConflictLogSchema
> (replica_identity, remote_tuple, local_conflicts), the total in-memory
> formed row size is bounded by ~1.8GB, ensuring it stays within
> PostgreSQL's 2GB row size limit (assuming local_conflicts contains at
> most one local tuple, which holds for all resolvable conflicts today).
>

A few comments:

1)
+ * bounded by ~1.8GB, ensuring it stays within PostgreSQL's 2GB row size

The comments mention a “2GB row limit,” but it is not clear to me
where this limit is derived from. The doc does not mention such a
limit:
https://www.postgresql.org/docs/19/limits.html

2)
Also going by worst case scenario as you mentioned above, the total
in-memory row size could go upto 1.8GB; won't that fail in
heap_form_tuple() in insert_conflict_log_tuple(). heap_form_tuple()
does palloc0 for total length which will MaxAllocSize limit of 1GB in
such a case.

3)
+#include "utils/jsonfuncs.h"

It compiles without this.

4)
+ <type>json</type>
+ <type>JSON</type>

We can make the case same for all. There are a few such references. I
think it is 'json' if we want to say it as pg-datatype and 'JSON' as
a generic term.

5)
+ <row>
+ <entry><literal>has_omitted_values</literal></entry>
+ <entry><type>boolean</type></entry>
+ <entry>Indicates whether any value in this row was too large to be
+ recorded, and was replaced by a marker. See below.</entry>
+ </row>

Suggestion:
Indicates whether any value in this row was too large to be recorded
and was replaced by an <literal>omitted</literal> marker containing
its length in bytes.

thanks
Shveta