| From: | Greg Burd <greg(at)burd(dot)me> |
|---|---|
| To: | pgsql-hackers(at)lists(dot)postgresql(dot)org |
| Cc: | Nikhil Kumar Veldanda <veldanda(dot)nikhilkumar17(at)gmail(dot)com>, Michael Paquier <michael(at)paquier(dot)xyz>, Japin Li <japinli(at)hotmail(dot)com>, Dharin Shah <dharinshah95(at)gmail(dot)com>, Manu <manuelreyesbravo(at)gmail(dot)com> |
| Subject: | Re: ZSTD TOAST compression, and an extensible compression method encoding |
| Date: | 2026-10-27 11:57:37 |
| Message-ID: | lHIOcamwvyJPNB1Fp-A0tEalU92hYzfEeU5iF0RPHvXzfcKSg2vkKnxw3bzO4_Gw5vIX9bPZVQBY4e1vUJDpPFjWFVeFdMfGv-9MPLjLuJQ=@burd.me |
| Views: | Whole Thread | Raw Message | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-hackers |
Okay, v12 on master (065dc52f1a4).
Fixes since v11,
- lz4-without-zstd did not compile (dict_fetch() was under USE_ZSTD but
called from the lz4 path); it is now shared, and an lz4-only build
passes make check.
- compression_zstd_dict used SET COMPRESSION lz4, which fails on a
zstd-only build; it uses pglz for that case now.
- automatic evolution (0005) swallowed errors, a query cancel included,
and could leave a half-done dict_install(). It now runs in an
internal subtransaction and re-throws a cancel, so an error rolls back
cleanly and a cancel stops ANALYZE.
- the per-backend dictionary caches held one entry, so evolution's
dictionary switching re-fetched and re-digested per value; they are
now a 4-entry LRU, and the held-out evaluation digests each candidate
once instead of once per value.
0001-0003 are unchanged (Nikhil's zstd method and the extensible
encoding).
0004 adds column-local TOAST compression dictionaries for zstd and lz4.
The catalog, the lifecycle, containment, retirement and training are
codec-independent; only three calls differ per codec, behind a small
DictCompressRoutine. lz4 has no trainer of its own and its headers point
at zstd's ZDICT, so an lz4 dictionary is the same trained bytes. A
dictionary helps once a value crosses TOAST_TUPLE_THRESHOLD; at about 3 kB
it takes lz4 from roughly 1.9x to 2.5x on docs and 3.6x to 4.8x on json.
0005 adds automatic dictionary evolution: opt-in per column, run inside
ANALYZE on the block sample it already has. It retrains only when the
data has grown past a threshold since the last training and a cooldown has
elapsed, trains a candidate on 80% of the sample, and adopts it only if it
compresses the held-out 20% at least a margin better than the current
dictionary. Nothing runs on the INSERT/UPDATE path; the write path reads
the current dictionary OID from a relcache cache. A per-column cap bounds
how many dictionaries accumulate. It is codec-agnostic: a codec adds only
train() and a compressed-size estimate.
Measured on drifting data, the stored size of the drifted values: plain
zstd 868 B, a stale dictionary 892 B (worse than none, as expected), an
evolved dictionary 821 B. Evolution recovers the dictionary's advantage
automatically after drift.
0005 depends only on 0004 and can be held back if the series is too large
to carry at once.
best.
-greg
| Attachment | Content-Type | Size |
|---|---|---|
| v12-0005-Add-automatic-dictionary-evolution-during-ANALYZ.patch | text/x-patch | 54.9 KB |
| v12-0001-Decode-the-compression-method-in-toast_external_.patch | text/x-patch | 6.0 KB |
| v12-0002-Allow-more-than-four-TOAST-compression-methods.patch | text/x-patch | 30.8 KB |
| v12-0003-Add-zstd-as-a-TOAST-compression-method.patch | text/x-patch | 83.5 KB |
| v12-0004-Add-TOAST-compression-dictionaries-for-zstd-and-.patch | text/x-patch | 124.3 KB |
| From | Date | Subject | |
|---|---|---|---|
| Previous Message | Greg Burd | 2026-10-27 11:53:23 | Re: ZSTD TOAST compression, and an extensible compression method encoding |