Re: Proposal: Conflict log history table for Logical Replication

On Thu, Oct 22, 2026 at 5:23 PM Nisha Moond <nisha(dot)moond412(at)gmail(dot)com> wrote:
>
> On Fri, Oct 16, 2026 at 8:03 PM Nisha Moond <nisha(dot)moond412(at)gmail(dot)com> wrote:
> >
> > Attached is v82, which addresses the comments on v81:
> >
>
> We have not yet documented that, in rare cases, the subscriber can
> still fail with the 1GB error when recording a conflict.
>
> The 64kB cap on replica identity values is checked after the key
> value's output function has produced its text, so it limits what we
> store, not what we render. Normally that's fine: the publisher already
> rendered the same value to send it, which keeps the text well within
> the limit. That doesn't hold in a few rare cases:
> - the key column has a different data type on the subscriber (e.g.
> text published into a numeric[] column);
> - binary = true, where the publisher never produced the text;
> - the apply worker's bytea_output or quote_all_identifiers differs
> from the publisher's.
>
> In these cases a small stored value can render to more than 1GB, and
> the apply worker errors out.
>

The attached file has standalone steps to reproduce each of the above
cases where recording a conflict in CLT can still hit the 1GB limit on
the subscriber.

These remaining cases are not specific to the CLT. The same problem
already exists wherever we call a type's output function to put a
value into a message.
On PG17 and older branches too, the subscriber already hits OOM on a
key violation. The apply worker fails while building the
unique-violation message: _bt_check_unique() ->
BuildIndexValueDescription() -> record_out()

Below is a test case for binary = true that reproduces this on PG17.
All four cases could fail the same way on PG17 when the subscriber
hits a key violation.

[PG17 binary=true testcase]
---------------------------------
-- [PUB] and [SUB]
DO $$ BEGIN
CREATE TYPE ri_n0 AS (f text);
FOR i IN 1..24 LOOP
EXECUTE format('CREATE TYPE ri_n%s AS (f ri_n%s)', i, i - 1);
END LOOP;
END $$;

-- [PUB]
CREATE TABLE kv1 (k ri_n24);
CREATE PUBLICATION pub_kv1 FOR TABLE kv1;

-- [SUB] the key exists locally
CREATE TABLE kv1 (k ri_n24 PRIMARY KEY);
DO $$ DECLARE r text := format('ROW(%L)::ri_n0', repeat('"', 60));
BEGIN
FOR i IN 1..24 LOOP r := format('ROW(%s)::ri_n%s', r, i); END LOOP;
EXECUTE format('INSERT INTO kv1 VALUES (%s)', r);
END $$;
CREATE SUBSCRIPTION sub_kv1 CONNECTION 'port=9943 dbname=postgres'
PUBLICATION pub_kv1 WITH (copy_data = false, binary = true);

-- [PUB] same key, sent in binary
DO $$ DECLARE r text := format('ROW(%L)::ri_n0', repeat('"', 60));
BEGIN
FOR i IN 1..24 LOOP r := format('ROW(%s)::ri_n%s', r, i); END LOOP;
EXECUTE format('INSERT INTO kv1 VALUES (%s)', r);
END $$;

-- [SUB] expect: ERROR: out of memory, instead of the unique violation:
ERROR: out of memory
DETAIL: Cannot enlarge string buffer containing 1073741822 bytes by 1
more bytes.
CONTEXT: processing remote data for replication origin "pg_16614"
during message type "INSERT" for replication target relation
"public.kv1" in transaction 761, finished at 0/1590948
~~~~

--
Thanks,
Nisha