Re: Proposal: Conflict log history table for Logical Replication

On Sun, Aug 30, 2026 at 8:43 PM Dilip Kumar <dilipbalaut(at)gmail(dot)com> wrote:
>
> On Wed, Aug 26, 2026 at 8:01 PM Dilip Kumar <dilipbalaut(at)gmail(dot)com> wrote:
> >
> Updated version of 0002, based on offlist testing by Nisha, revealed
> that the assumption that a column's maximum size could become 6x its
> original size during JSON conversion is incorrect. One edge case is a
> column of type array (int[]). A huge array but mostly empty(all NULL),
> can have a small storage size but produce a much larger JSON since
> each element is serialized. For example
> - A 4096 byte text column becomes ~24KB json (6x), which stays under
> the budget even accumulated across all columns (24,554 × 1600 × 3 ≈
> 118 MB).
> - But a 4096 byte all-NULL int[] column can produce ~480KB(120x) of
> json in a edge case.
> Based on this, Amit suggested a offlist POC of the patch which Nisha
> and I further modified. It still needs more review, testing, and logic
> validation, but I am sharing it here so we can review and provide
> feedback.
>

Thanks Dilip for the patches.

Patch v73-002:
While testing the column size limits, I found a case where storing a
very large jsonb value is possible, but trying to describe or process
the value hits the 1GB string limit.

Testcase:
-- BOTH nodes
CREATE TABLE t1 (a int PRIMARY KEY, j jsonb);

-- PUB
INSERT INTO t1 (a) VALUES (1);

-- SUB
-- after sync, update the local value so the row's origin differs:
UPDATE t1 SET j = (SELECT ('['||string_agg('1e131071', ',')||']')::jsonb
FROM generate_series(1,8300)) WHERE a = 1;

-- PUB: trigger update_origin_differs
UPDATE t1 SET a = 1 WHERE a = 1;
~~~

On sub, after the update, the value in the j column hits a legitimate
limit, we cannot even select or replicate it:
postgres=# select j from t1;
ERROR: string buffer exceeds maximum allowed length (1073741823 bytes)
DETAIL: Cannot enlarge string buffer containing 1073627135 bytes by
131072 more bytes.

-- So the j column is effectively not processable at this size. The
exponent (1e131071) fits in numeric, but flattening it into jsonb
(with ~8300 repetitions) exceeds the limit.

But if such a value already exists as a local tuple on the
subscriber(above case), the apply worker can hit this limit while
trying to log the local tuple during conflict handling (including the
CLT path when enabled).

I'm not sure whether it is acceptable for conflict logging to hit this
limit and break the apply worker, especially since we say that the
logged values are trimmed or omitted.

--
Thanks,
Nisha