Re: ZSTD TOAST compression, and an extensible compression method encoding

Here is v10, with the change I promised in the last message and two new
patches.

0001 Decode the compression method in toast_external_info_get()
0002 Allow more than four TOAST compression methods
0003 Add zstd as a TOAST compression method (context reuse)
0004 Add zstd compression dictionaries with a column-local lifecycle
0005 Abstract the dictionary codec behind a routine struct
0006 Add lz4 compression dictionaries

0001-0004 are v9 with one change: the 0003 commit message now reads, per
my last message,

For whole-value reads and small (4kB) values this saves a few percent;
it rises to about 40% for small slices and prefixes of large (32kB)
values, where the allocation dominates the decode.

0005 and 0006 are new. Rebased on master (1910daf9831); cfbot is green.

0005, abstracting the codec. Reviewing 0004 it became clear the whole
dictionary machinery (the pg_toast_dictionary catalog, column ownership,
containment, retirement, training with ZDICT) is codec-independent. Only
three calls differ per codec: compress a value with a dictionary,
decompress a whole value, decompress a prefix. 0005 puts those behind a
DictCompressRoutine struct dispatched on the datum's method byte, so a
second dictionary-aware codec is three functions plus a case. No behavior
change for zstd; it only opens the seam.

0006, lz4 dictionaries. lz4 has no trainer of its own; its headers point
at zstd's ZDICT, which 0004 already uses, so a trained lz4 dictionary is
just bytes stored in pg_toast_dictionary exactly like a zstd one, and the
lifecycle is reused unchanged. 0006 adds one DictCompressRoutine for lz4
(LZ4_loadDict + LZ4_compress_fast_continue, LZ4_decompress_safe_usingDict)
and the lz4_dict method. Training still needs zstd's builder, so an
lz4_dict column wants a server built with both; without a dictionary it
behaves like plain lz4.

lz4 dictionaries earn their place. Per-value CLI numbers, each document
compressed on its own against a 64kB ZDICT dictionary:

~1kB structured docs: lz4 2.0x -> 3.6x (+80%), zstd 2.9x -> 5.2x
~2-6kB structured docs: lz4 2.8x -> 3.8x (+33%), zstd 5.4x -> 6.0x

At the small sizes where TOAST dictionaries help most, lz4 nearly doubles,
and lz4+dict (3.6x) beats plain zstd (2.9x) while keeping lz4's much lower
decompression cost. The proportional gain is largest for small values and
fades as values grow, which matches the zstd dictionary numbers from my
last message.

I kept 0005 and 0006 separate so the abstraction reads as a no-op refactor
and lz4 as self-contained. If the series is already too large, I'm happy
to hold 0006; it does not change 0001-0005.

One thing worth settling: automatic dictionary evolution. Training is
manual today (pg_train_toast_dictionary). As a column accumulates data,
when should a dictionary be rebuilt, on what sample, and how do we tell a
new dictionary is actually better, all without touching the INSERT/UPDATE
fast path? I have a design with the following shape:

- Opt-in per column (an attribute option, default off), evaluated during
ANALYZE, which already has a block sample. No new scheduler; it rides
the autovacuum ANALYZE cadence.
- Trigger on a multiplicative row-growth ratio since the dictionary was
trained, plus a cooldown, so the retrain interval grows with the table.
- Decide "better" by a held-out size test: train a candidate on 80% of
the sample, compress the other 20% with both the current and candidate
dictionary, adopt only on a margin win (a GUC, say 5%). This runs in
the maintenance process, never on the write path.
- Cache the current dictionary OID in the relcache so the fast path is a
field read, not a catalog probe per value. This also resolves the XXX
already in toast_dictionary_current().
- Bound pg_toast_dictionary with a per-column cap on retained
dictionaries; when hit, decline and warn rather than force a rewrite.
Retirement stays rewrite-bound as it is now; cheaper VACUUM-driven
retirement is a harder follow-up, because proving a dictionary
unreferenced needs a full attribute scan.
- All of it is codec-agnostic: a codec supplies only train(samples) and a
compressed-size estimate, which is the seam 0005 adds.

I can post the full design, or turn it into a patch once the policy is
agreed. It can equally wait until the manual lifecycle lands; I don't want
to widen the series anymore before 0001-0004 are settled, but someone was
bound to ask about this aspect of the solution.

best.

-greg