| From: | Dilip Kumar <dilipbalaut(at)gmail(dot)com> |
|---|---|
| To: | vignesh C <vignesh21(at)gmail(dot)com> |
| Cc: | shveta malik <shveta(dot)malik(at)gmail(dot)com>, Amit Kapila <amit(dot)kapila16(at)gmail(dot)com>, "Zhijie Hou (Fujitsu)" <houzj(dot)fnst(at)fujitsu(dot)com>, Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com>, saurabh singh <saurabh(dot)singh214(at)gmail(dot)com>, Robert Haas <robertmhaas(at)gmail(dot)com>, Peter Smith <smithpb2250(at)gmail(dot)com>, Nisha Moond <nisha(dot)moond412(at)gmail(dot)com>, Bharath Rupireddy <bharath(dot)rupireddyforpostgres(at)gmail(dot)com>, PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org> |
| Subject: | Re: Proposal: Conflict log history table for Logical Replication |
| Date: | 2026-08-30 15:13:07 |
| Message-ID: | CAFiTN-sdedotL28eFTeNX8_zVpn2o=u-SJ+t4W_wrPx_syAA6g@mail.gmail.com |
| Views: | Whole Thread | Raw Message | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-hackers |
On Wed, Aug 26, 2026 at 8:01 PM Dilip Kumar <dilipbalaut(at)gmail(dot)com> wrote:
>
Updated version of 0002, based on offlist testing by Nisha, revealed
that the assumption that a column's maximum size could become 6x its
original size during JSON conversion is incorrect. One edge case is a
column of type array (int[]). A huge array but mostly empty(all NULL),
can have a small storage size but produce a much larger JSON since
each element is serialized. For example
- A 4096 byte text column becomes ~24KB json (6x), which stays under
the budget even accumulated across all columns (24,554 × 1600 × 3 ≈
118 MB).
- But a 4096 byte all-NULL int[] column can produce ~480KB(120x) of
json in a edge case.
Based on this, Amit suggested a offlist POC of the patch which Nisha
and I further modified. It still needs more review, testing, and logic
validation, but I am sharing it here so we can review and provide
feedback.
--
Regards,
Dilip Kumar
Google
| Attachment | Content-Type | Size |
|---|---|---|
| v73-0002-Don-t-record-oversized-values-in-the-conflict-lo.patch | application/octet-stream | 32.9 KB |
| v73-0001-Implement-the-conflict-insertion-infrastructure-.patch | application/octet-stream | 81.4 KB |
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Tom Lane | 2026-08-30 15:33:43 | Re: remove_useless_joins vs. bug #19560 |
| Previous Message | Nathan Bossart | 2026-08-30 14:26:38 | Re: Add starelid, attnum to pg_stats and leverage this in pg_dump |