Re: ZSTD TOAST compression, and an extensible compression method encoding

Hi Nikhil,

I've been reviewing v7 and it looks solid to me. Full disclosure: Nikhil
and I work together at AWS and used to be on the same team, which is where
this patch set started, so I'm a fan and also biased.

So, I applied v7 to master as of 54635e6892f and everything worked as
expected.

Two points on v7:

0001: For an external value that isn't compressed, the new compress_method
field comes back as 0, which happens to mean pglz. Anyone who reads the
field without first checking VARATT_EXTINFO_IS_COMPRESSED() will think
the value is pglz-compressed. I'd have toast_external_info_get() fill in
the size and the compressed flag as well, and set compress_method to
TOAST_INVALID_COMPRESSION_ID when there's no compression. That's
essentially what Michael and wenhui suggested, and it would make 0001
useful on its own this cycle.

0003: You held off on Manu's context reuse because the WAL code used the
one-shot zstd calls. That's no longer true: Michael's 857340b0263 and
ce3387b74b7 made WAL keep its zstd contexts for the life of the backend.
Given Manu's numbers (slices 23-34% faster, full reads of 32kB values
11-13% faster), I'd fold something similar to his change into 0003.

I know, I know... but why not talk about them?

Dictionaries
------------

You and Michael have both put dictionaries out of scope until plain zstd is
in, and that is the right order for commits. Michael's concern upthread
still stands:

> What worries me the most is the amount of work between the on-disk
> data and the backend in terms of retention of past dictionnaries.

The first thread stopped on the same problem. Robert's objections to v11's
pg_depend tracking [1], Andres's point about undroppable dictionaries
accumulating [2], and the per-insert cost of v13's flatten_datum [3] were
deferred rather than answered. 0002 spends the last free value of the method
bits on the grounds that dictionary support would be another method ID in the
long form. It seems worth showing before 0002 lands that a dictionary method
with a workable lifecycle fits that form.

The attached v20261015-0004 is a patch on top of v7 that does that, following
Robert's suggestion in [4]: dictionary-compressed datums exist only in the
column that owns the dictionary, and are decompressed whenever they would be
stored anywhere else. 0001-0003 of the attached series are your v7, unchanged
apart from the rebase onto aa7b9cf132b.

Ownership. A new catalog, pg_toast_dictionary (oid, tdrelid, tdattnum,
tdcurrent, tddict), whose rows belong to one column as pg_statistic rows do.
DROP TABLE removes them, next to RemoveStatistics(). DROP COLUMN does not: the
dropped column's values remain in the heap, and whole-row references and
REPLICA IDENTITY FULL still read them, until the next rewrite retires the
dictionaries. There are no pg_depend entries.

Containment. heap_fill_tuple() sets HEAP_HASEXTERNAL on a tuple holding a
zstd_dict datum, as it does for a TOAST pointer. Every path that already
flattens such tuples (toast_flatten_tuple() and its variants, expanded records,
the catcache, heap_update()'s need_toast) therefore reaches the datum, and
detoast_external_attr() decompresses it. The few places that test
VARATT_IS_EXTERNAL() on a single datum (index_form_tuple, brin_form_tuple,
tstoreReceiver, PL/pgSQL non-atomic assignment) also test
VARATT_IS_COMPRESSED_DICT().

On store, toast_tuple_init() keeps a zstd_dict datum compressed only if the
target column owns the dictionary and the dictionary is current; otherwise it
decompresses it and the target column's method applies. Values compressed with
pglz, lz4 or plain zstd are copied as before, so the CTAS slowdown seen with
v13's flatten_datum applies only when the source column uses a dictionary.

Retirement. pg_train_toast_dictionary(rel, col, dict_size) trains from the
column's values and makes the new dictionary current; the previous one stays
readable. The next rewrite (VACUUM FULL, CLUSTER, REPACK, or a rewriting ALTER
TABLE) recompresses values that use a non-current dictionary, and
finish_heap_swap() then deletes it. Dictionaries of a column switched to
another method, or dropped, are retired the same way. Rewrites already hold
AccessExclusiveLock. Training takes ShareUpdateExclusiveLock, as ANALYZE does.

Upgrade. pg_upgrade keeps the heap files, which embed dictionary OIDs, so
binary-upgrade dumps restore each dictionary with its OID and physical column
number, including those of switched or dropped columns that have not been
rewritten yet.

On disk, a zstd_dict datum is the v7 long header plus the dictionary OID:

va_header | va_tcinfo (method bits 11) | va_cmid = 4 | dictid | frame

so 0002 needs no change for it.

I've not yet toyed with the following, maybe you have ideas?

- Training is an explicit call over the first 10000 rows. Training from
ANALYZE's sample is the natural next step.
- toast_dictionary_current() does an index probe per compressed value,
and the CDict/DDict cache holds one entry each way. Both can move to
the relcache entry if profiling shows a need.
- Prefix slices of dictionary values decode the whole value, which costs
nothing extra below zstd's 128kB block size.
- No compression level setting.

Hannu's encoding-context proposal [5] is broader. I'd expect the ownership and
containment rules here to apply to it as well.

If this matches what you had in mind for dictionaries, I'd suggest carrying it
as a later patch in your series, so the long-form discussion has a concrete
dictionary user to point at. I can rebase it on v8.

best.

-greg

[1] https://postgr.es/m/CA+Tgmob1Ux7R2_0mbZmTbsVmUda04X1AdHGjpu6FkZfdbztugQ@mail.gmail.com
[2] https://postgr.es/m/wjrd7ubqm4sq6t4ddv2ae7xaf26vcf4z6i7nftag3e6zexqmyc@yjilk5fsqcfh
[3] https://postgr.es/m/CAFAfj_Fee7zQ7YNnJObLMqVF=MpFwVHszzPJ-ZWkiiade04SAw@mail.gmail.com
[4] https://postgr.es/m/CA+TgmoY8uCKFvVYDQsLbfhY3JYrduXE7WHi75hmfTWxjWioaaw@mail.gmail.com
[5] https://wiki.postgresql.org/wiki/Pg_encoding_context