Re: ZSTD TOAST compression, and an extensible compression method encoding

Hi Greg

Please take a look at a generalized design for the compression dictionary

In email: [PATCH v1] Generic Datum Encoding and Compression Contexts
(pg_encoding_type, pg_encoding_context)

I have used this myself in dictionary compression for zstd and lz4 and
for keeping Apache Avro schemas for JSON and JSONB encodings and
defining cascaded Avro+zstd

And also I used the same for storing quantization tables for pgvector
int8 and other quantizations

This could also be used for other "encoding" tasks, like having a
public key for writes and a more restricted private key for reads,
etc.

The main point is that having a generic facility for these kinds of
tasks would benefit many use cases.

Cheers,
Hannu

On Sun, Oct 25, 2026 at 5:34 PM Greg Burd <greg(at)burd(dot)me> wrote:
>
> Hello, here is v11.
>
> Two changes on top of v10, both in the dictionary area, plus a rebase on master
> (065dc52f1a4). cfbot is green on all platforms. It does not touch 0003 or the
> decode paths from the measurement thread.
>
> 0001 Decode the compression method in toast_external_info_get()
> 0002 Allow more than four TOAST compression methods
> 0003 Add zstd as a TOAST compression method
> 0004 Add TOAST compression dictionaries for zstd and lz4
> 0005 Add automatic dictionary evolution during ANALYZE
>
> 0001-0003 are unchanged.
>
> Some callouts.
>
> 0004 folds the former abstraction and lz4-dictionary patches into one.
>
> The split was artificial: the catalog, the column-local lifecycle, containment,
> retirement and training are codec-independent, and only three calls differ per
> codec (compress with a dictionary, decompress, decompress a prefix). So the
> dictionary patch now carries both zstd_dict and lz4_dict from the start, with
> the codec behind a small DictCompressRoutine. lz4 has no trainer of its own
> and its headers point at zstd's ZDICT, so an lz4 dictionary is the same trained
> bytes; only the compress/decompress calls change. For small, similar values,
> which is where TOAST dictionaries help most, a dictionary roughly doubles lz4
> (about +80% on ~1kB structured documents) and brings lz4+dict above plain zstd
> while keeping lz4's lower decompression cost.
>
> 0005 automatic dictionary evolution, the open question from last time. (new)
>
> A trained dictionary ages as a column's data drifts. This lets ANALYZE
> retrain, opt-in per column (an attribute option, default off), on the block
> sample ANALYZE already has, and never on the write path:
>
> - Gates. Retrain only when the live row count has grown by a factor
> since the current dictionary was trained, is past a floor, and the
> current dictionary is older than a cooldown. A per-column cap bounds
> how many dictionaries accumulate; at the cap it declines with a LOG
> and suggests a rewrite, since retirement is still rewrite-bound.
> - Adopt only on a win. Train a candidate on 80% of the sample,
> compress the held-out 20% with both the current and the candidate
> dictionary, and install the candidate only if it is at least a margin
> smaller on data it was not trained on. The first dictionary is
> adopted unconditionally.
> - Off the fast path. All of this runs in the ANALYZE process. The
> write path only reads the current dictionary OID, now cached in the
> relcache (dict_install invalidates it), so a bulk load no longer
> probes the catalog per value.
> - Codec-agnostic. A codec adds only train() and a compressed-size
> estimate, so zstd and lz4 share the whole policy.
>
> Two pg_toast_dictionary columns feed the gates: tdtrainrows (the reltuples
> estimate at training time) and tdtraintime.
>
> Measured on drifting data: load one distribution, train, drift to a different
> distribution, ANALYZE, rewrite, and compare the stored size of the drifted data
> under three policies:
>
> policy avg stored ratio
> plain zstd (no dictionary) 868 B 3.94
> stale dictionary (old data, kept) 892 B 3.83
> evolved dictionary (retrained) 821 B 4.16
>
> The point is the middle row: after drift a stale dictionary is worse than no
> dictionary, because it mis-predicts the new data; evolution retrains
> automatically and recovers the advantage (821 vs 892, about 8% smaller). The
> decision is visible in the log at DEBUG1:
>
> dictionary evolution for "v": heldout=6000 dictsize=65536
> cur=5179154 cand=4910999 gain=5.2% adopt=1
>
> When the distribution does not drift, the candidate does not beat the incumbent
> by the margin, so nothing is installed and the dictionary count does not move.
> The held-out test is a compression-ratio proxy, not an end-to-end TOAST
> measurement; I think that is the right first cut, and the margin absorbs the
> slack.
>
> 0005 depends only on 0004, so if the series is more than the thread wants to
> carry at once I can hold it back and post it separately later.
>
> The auto-training/retraining aspect is likely going to elicit a lot of good
> design feedback and differing opinions, but I'd like those who participate in
> the discussion about that to first acknowledge that the fundamental idea of
> auto-retraining/evolution of dictionaries is an idea that they feel should be
> in or out of this kind of solution explicitly and up front if possible. I do
> think it is a good idea and should be in, but there should be functions to
> manage dictionaries manually and a way to enable/disable the automatic piece.
>
> my $0.02, fwiw. best.
>
> -greg