| From: | Greg Burd <greg(at)burd(dot)me> |
|---|---|
| To: | Hannu Krosing <hannuk(at)google(dot)com> |
| Cc: | pgsql-hackers(at)lists(dot)postgresql(dot)org, Manu <manuelreyesbravo(at)gmail(dot)com>, Nikhil Kumar Veldanda <veldanda(dot)nikhilkumar17(at)gmail(dot)com>, Michael Paquier <michael(at)paquier(dot)xyz>, Japin Li <japinli(at)hotmail(dot)com>, Dharin Shah <dharinshah95(at)gmail(dot)com>, Robert Haas <robertmhaas(at)gmail(dot)com> |
| Subject: | Re: ZSTD TOAST compression, and an extensible compression method encoding |
| Date: | 2026-11-22 16:52:48 |
| Message-ID: | 179535923952.3942862.9182006155358054736@burd.me |
| Views: | Whole Thread | Raw Message | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-hackers |
Hi all,
v21 is attached, rebased onto 3ed10f282f9.
Where things stand: Hannu reviewed v20 and found nothing further, and
0001-0003 have had no new findings for several versions. Since then the
review has turned to whether a dictionary-compressed value can outlive
the dictionary it needs, and v21 is about that.
The rule the design rests on is that such a value never leaves the
column that owns its dictionary in compressed form. heap_fill_tuple()
sets HEAP_HASEXTERNAL on a tuple that holds one, so the paths that
already make a tuple self-contained for TOAST pointers (composite
datums, tuplestores, holdable cursors, PL/pgSQL records, the syscache,
index tuples) decompress it too. That costs nothing for anyone not
using dictionaries: under callgrind, heap_fill_tuple() runs the same
number of instructions for values that are not compressed in-line, and
about five more per compressed value, which is 0.03% of an INSERT ...
SELECT.
Three questions came up about where that rule could break, and v21
changes 0004 to answer them:
- A PL/pgSQL function could keep a value in a record variable after its
dictionary was gone. Within a transaction, a function fills a record
variable without flattening it, which is safe for TOAST pointers
because a table cannot lose its TOAST data before the transaction
ends. A dictionary can go sooner: rolling back the subtransaction
that trained it (an EXCEPTION block is enough) deletes its row, and so
does a rewrite of the table. v20 then failed with "compression
dictionary N does not exist", and since a later dictionary may get the
same OID, it could have decoded with the wrong one. The record path
now always decompresses dictionary values, and the regression test
covers the rollback, the rewrite and DROP TABLE.
- Other places could miss the rule. I went through every place in core
and contrib that tests a single value with VARATT_IS_EXTERNAL() and
found no other gap; the list is in notes.txt. So that new code cannot
miss it the same way, the test is now one predicate,
VARATT_IS_EXTERNAL_OR_DICT(), which every such place uses.
- An extension type that embeds other varlenas could copy one byte for
byte. Core cannot see a value once it is inside another type's bytes,
so this cannot be enforced, only documented: the user-defined types
chapter now says such a type must detoast what it embeds, as it
already must for TOAST pointers. hstore, ltree, intarray, jsonb,
pgvector and PostGIS all do.
Changes since v20:
0004: dictionary values in PL/pgSQL record variables are decompressed
when the record is filled (toast_flatten_tuple_dict()), with tests
for subtransaction rollback, rewrite and DROP TABLE
0004: VARATT_IS_EXTERNAL_OR_DICT() replaces the paired tests
0004: xtypes.sgml documents the rule for container types
0002: a comment in verify_heapam.c rewrapped for pgindent
0001, 0003 and 0005 are unchanged apart from the rebase. The audit, the
extension survey and the instruction counts are in the attached
notes.txt created with a bit of LLM help.
-greg
| Attachment | Content-Type | Size |
|---|---|---|
| notes.txt | text/plain | 6.8 KB |
| v21-0005-Add-automatic-dictionary-evolution-during-ANALYZ.patch | text/x-patch | 103.0 KB |
| v21-0001-Decode-TOAST-pointers-fully-in-toast_external_in.patch | text/x-patch | 13.0 KB |
| v21-0004-Add-TOAST-compression-dictionaries-for-zstd-and-.patch | text/x-patch | 191.0 KB |
| v21-0002-Allow-more-than-four-TOAST-compression-methods.patch | text/x-patch | 52.8 KB |
| v21-0003-Add-zstd-as-a-TOAST-compression-method.patch | text/x-patch | 97.1 KB |
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Zhijie Hou | 2026-11-22 17:46:12 | Fix LogStandbySnapshot() race during logical decoding activation |
| Previous Message | Greg Burd | 2026-11-22 16:35:34 | Re: Let an ordering index scan hand its ORDER BY value to the target list |