Re: Proposal: Conflict log history table for Logical Replication

On Wed, Aug 26, 2026 at 5:54 PM Dilip Kumar <dilipbalaut(at)gmail(dot)com> wrote:
>
> The updated patch implements this suggestion and also fixes the issues
> Shveta raised. And changed CONFLICT_MAX_VALUE_SIZE to 16KB and added
> the comments, With CONFLICT_MAX_VALUE_SIZE set to 16kB, even a table
> with the maximum supported number of columns (MaxHeapAttributeNumber =
> 1600) undergoing the worst-case 6x JSON escape expansion will produce
> a serialized JSON datum of at most ~150MB, safely below PostgreSQL's
> 1GB varlena limit. Furthermore, across all three JSON columns in
> ConflictLogSchema (replica_identity, remote_tuple, local_conflicts),
> the total in-memory formed row size is bounded by ~460MB, ensuring it
> stays comfortably within PostgreSQL's 1GB (MaxAllocSize) in-memory
> row/tuple size limit.
>

Hi,
For patch v72-002:

After digging a bit into how JSON is generated for different data
types, the 6x size assumption does not seem to hold for all cases.

One edge case is an int[] (or any type array) column. A large array
with mostly/all NULL values can have a small storage size but produce
a much larger JSON representation, since each element is serialized.

For example -
- A 4096 byte text column becomes ~24KB(6x) json, which stays under
the budget even accumulated across all columns (24,554 × 1600 × 3 ≈
118 MB).
- But a 4096 byte all-NULL int[] column can produce ~480KB(120x) of json.
For a 6-D array such as:
column = array_fill(NULL::int,ARRAY[130500,1,1,1,1,1])

-- Each null element of array is converted into json as
"[[[[[null]]]]]," = ~15 chars
-- The json rendered becomes = 130500 * 15 = 1957500 chars = ~1.9MB

I'm able to reproduce the error in the apply loop with a test case[1]
using 300 such array columns in the table.

58843] LOG: conflict detected on relation "public.clt_big300":
conflict=update_origin_differs
[58843] DETAIL: Conflict details are logged to the conflict log
table: pg_conflict.pg_conflict_log_16390
[58843] CONTEXT: processing remote data for replication origin
"pg_16390" during message type "UPDATE" for replication target
relation "public.clt_big300" in transaction 716, finished at
0/018A2780
[58843] ERROR: invalid memory alloc request size 1174505445

Several other column types can lead to the same issue: an array of
composites, a jsonb column containing numerics with large exponents
(jsonb stores numerics in binary, so its storage size says nothing
about its output size), or a type with an explicit CREATE CAST ... AS
json, whose output size is unrelated to the value being cast.

I think the issue is in this check:
````
+ if (toast_raw_datum_size(values[i]) - VARHDRSZ >
+ CONFLICT_MAX_VALUE_SIZE)
+ {
+ found_oversized = true;
+ break;
+ }
````
We are comparing the storage size and not the actual json size which
will be eventually stored in clt. The expansion factor depends on the
value's type and shape, not just its size. So there doesn't seem to be
a storage-size cap that is safe for all such cases.

Shouldn't we compare the JSON size after converting each column?
~~~

[1] Simple Reproducer -
- Assuming a publisher-subscriber setup exists for the table with CLT
configured.

Table, on both nodes:
DO $$
DECLARE cols text;
BEGIN
SELECT string_agg(format('c%s int[]', i), ', ') INTO cols FROM
generate_series(0,299) i;
EXECUTE format('CREATE TABLE clt_big300 (a int PRIMARY KEY, %s)', cols);
END $$;

-- pub
INSERT INTO clt_big300 (a) VALUES (1);

Now make both sides large tuple.

-- On sub (local -> this becomes local_conflicts):
DO $$
DECLARE sets text;
BEGIN
SELECT string_agg(format('c%s = array_fill(NULL::int,
ARRAY[130500,1,1,1,1,1])', i), ', ')
INTO sets FROM generate_series(0,299) i;
EXECUTE format('UPDATE clt_big300 SET %s WHERE a = 1', sets);
END $$;

-- On pub: same statement (this becomes remote_tuple and triggers
update_origin_differs):
DO $$
DECLARE sets text;
BEGIN
SELECT string_agg(format('c%s = array_fill(NULL::int,
ARRAY[130500,1,1,1,1,1])', i), ', ')
INTO sets FROM generate_series(0,299) i;
EXECUTE format('UPDATE clt_big300 SET %s WHERE a = 1', sets);
END $$;
~~~

--
Thanks,
Nisha