Re: Proposal: Conflict log history table for Logical Replication

On Tue, Sep 1, 2026 at 4:03 PM shveta malik <shveta(dot)malik(at)gmail(dot)com> wrote:
>
Here is detailed analysis and summary of the problem and all the
alternatives we tried. The problem is that converting a
TupleTableSlot to JSON can result in rendered JSON text exceeding
PostgreSQL's MaxAllocSize limit (1 GB). When that happens, memory
allocation (palloc() or enlargeStringInfo()) throws a hard ERROR.

This issue is not unique to conflict logging:
1) pgoutput does not handle this: if a row's formatted textual
representation exceeds 1 GB in logicalrep_write_tuple(),
OidOutputFunctionCall() or palloc() fails with an unrecoverable hard
error.
2) For conflict logging, converting to JSON exacerbates the problem
because a) every byte below 0x20 expands into a 6-byte \uXXXX
sequence. b) JSONB values (e.g., repeated numeric expansions like
1e131071), arrays, and composites can be compact on disk but expand to
hundreds of megabytes or gigabytes when serialized to text. c)
user-Defined Types (UDTs): A compact binary UDT (e.g., a run-length
encoded vector) can expand into gigabytes inside its typoutput
function.
3) Other conflict logging extensions (e.g., pgactive and pgEdge) also
have this issue, and none of them handle it.

Because storing multi-gigabyte or hundreds-of-megabytes values in a
conflict log table is neither practical nor desirable for post-mortem
debugging, we have been exploring several options to gracefully handle
oversized attributes without erroring out. Below is a summary of the
options explored, along with their pros and cons.

Option 1: Attribute-Level Size Capping via ErrorSaveContext (escontext)
Enforce a reasonable threshold (e.g., 16 KB) per attribute during JSON
conversion. Pass an ErrorSaveContext to an extended serialization
function (datum_to_json_extended()). As serialization recurses through
arrays, composites, or JSONB containers, cumulative size is monitored.
If an attribute exceeds the limit, errsave() records the soft error
and returns (Datum) 0. The caller detects
SOFT_ERROR_OCCURRED(&escontext) and cleanly replaces that attribute in
the outer JSON tuple with an omission object: {"omitted": true,
"length": ...}.
Pros:
- Retains the full tuple structure and all other normal-sized
columns for analysis.
- Cleanly handles unbounded recursion in nested structures (arrays,
composites, and JSONB) by aborting early.
Cons:
- Does not protect against typoutput failures: standard PostgreSQL
output functions invoked via OidOutputFunctionCall() do not accept an
ErrorSaveContext parameter.
- If a UDT (or built-in bytea with a large payload in hex format)
exceeds 1 GB inside its typoutput function, palloc() throws a hard
ERROR before size checks run; escontext cannot intercept it.

Option 2: Record Only Replica Identity (RI) Columns Instead of Full Tuples
Instead of serializing entire remote and local tuples to JSON, only
serialize the Replica Identity key columns (typically Primary Key or
Unique Index attributes) identifying the conflicting row.
Pros:
- Drastically reduces conflict log storage overhead and serialization cost.
- For standard integer, UUID, or short text keys, the payload is
tiny and never approaches memory limits.
Cons:
- Does not fully eliminate the problem: Replica Identity can be
defined on a UDT or large composite key. If that key attribute expands
to > 1 GB in typoutput, conflict logging will still error out. In
short, this leaves us with the same problem as Option 1.

Option 3: Whitelist Only Fixed-Length / Safe Built-in Data Types in v1
In the initial version of conflict logging, only serialize columns
with guaranteed small, bounded types (e.g., fixed-length types like
int2, int4, int8, float4, float8, bool, date, timestamp, uuid). Any
varlena type, container, or UDT is automatically omitted without
invoking its output function.
Pros:
- Completely immune to palloc() 1 GB overflow by construction.
- Simple to implement with zero chance of erroring out.
- Safe baseline that can be incrementally expanded in future releases.
Cons:
- Could be restrictive: common types like text, varchar, jsonb, and
numeric are omitted even when their values
are just a few bytes (e.g., a 10-character varchar column).

Option 4: Wrap Attribute Serialization in PG_TRY() / PG_CATCH()
While serializing each attribute of the tuple to JSON, wrap the
conversion (specifically OidOutputFunctionCall()) inside an internal
subtransaction with a PG_TRY() / PG_CATCH() block:
Pros:
- Catches all hard errors, including 1 GB palloc() exhaustion inside
uncooperative typoutput functions, memory allocation failures, or
corrupted data.
- Enables full support for all data types (built-in, JSONB, varlena,
and UDTs) without risking apply worker retry loops.
- Does not require changing PostgreSQL's global typoutput function
signature to support escontext.
Cons:
- Using PG_TRY() and internal subtransactions adds management
overhead (though conflict logging is an exceptional path, not the main
transaction fast-path).
- Consumes Transaction IDs (subXIDs), though read-only in-memory
subtransactions are relatively lightweight.

In summary, only Options 3 and 4 are viable; while the first two
options reduce the error surface, they do not eliminate it entirely.

--
Regards,
Dilip Kumar
Google