Re: ZSTD TOAST compression, and an extensible compression method encoding

On Monday, October 26th, 2026 at 3:02 PM, Greg Burd <greg(at)burd(dot)me> wrote:
>
> On Sunday, October 25th, 2026 at 12:52 PM, Hannu Krosing <hannuk(at)google(dot)com> wrote:
>
Hi Hannu,

I read the Generic Datum Encoding and Compression Contexts patch. It is a
bigger idea than this one, and worth pulling apart where the two meet and
where they don't. Inline below.

> Please take a look at a generalized design for the compression
> dictionary [...] I have used this myself in dictionary compression for
> zstd and lz4 and for keeping Apache Avro schemas for JSON and JSONB
> encodings and defining cascaded Avro+zstd

Where we agree: a trained dictionary is just one case of "a value needs a
side artifact to be decoded." Your pg_encoding_context is that artifact
in the general form, and a zstd/lz4 dictionary is one kind of content for
it. The two designs even store nearly the same thing: an id, owning
state, and a bytea blob.

Where they differ is scope, and it is the scope that drove this design.
This series is about TOAST, specifically. The hard problem it solves is
not "store a dictionary," it is "know when a dictionary can be dropped
without a full table scan to prove no value still needs it." The answer
here is column-local ownership: a dictionary belongs to one (relid,
attnum) like a pg_statistic row, values compressed with it never leave
that column in compressed form (heap_fill_tuple marks them external so
every self-contained copy detoasts them), and a rewrite is what retires
the old one. That containment rule is the whole point, and it is TOAST
machinery, not general datum machinery; I don't see how to express it in a
facility that doesn't know about attcompression, detoast, and the rewrite
path.

We'll have to consider this, and other use cases if your generalized
method is going to work.

> And also I used the same for storing quantization tables for pgvector
> int8 and other quantizations [...] like having a public key for writes
> and a more restricted private key for reads, etc. The main point is
> that having a generic facility for these kinds of tasks would benefit
> many use cases.

This is the part I'd be careful with. Quantization tables, Avro schemas,
and key management each have their own lifecycle and their own "when can
this be dropped" answer, and those answers are not the same as a TOAST
dictionary's. A facility general enough to hold all of them tends to push
the lifecycle decision back onto each caller, which is exactly the part
that was hard to get right here. I'm not against the generic facility; I
am wary of making the TOAST feature wait on, or inherit the constraints
of, a design that also has to satisfy pgvector and key rotation.

> [from the follow-up] At least the underlying storage should allow
> multiple dictionaries for the same relation and moving to newer
> dictionaries or some heuristic for dictionary selection.

That part we already do, and it's the strongest point of overlap.
pg_toast_dictionary holds many dictionaries per column, each value carries
the 4-byte OID of the one it was compressed with, exactly one is current
(tdcurrent) and new writes use it, and 0005 adds the heuristic you
describe: ANALYZE trains a candidate and adopts it only when a held-out
test says it beats the incumbent. So the "multiple dictionaries, move to
newer ones, select by heuristic" shape is in the series today.

> It also would be nice to have an option for "retiring" a dictionary by
> - setting it read-only, so no new rows can use it
> - having some process re-encode old values (or [...] wait until
> partitions with such values are dropped)
> - then have a verify and drop function

This is close to what happens, under different names. Marking a
dictionary non-current is your "read-only": no new value uses it, old ones
still read. The re-encode step is a table rewrite (VACUUM FULL, CLUSTER,
REPACK), which recompresses every value with the current dictionary and
then drops the non-current ones. What the series does not have, and your
framing makes a fair case for, is a rewrite-free retirement: a verify-then-
drop that proves no live value references a dictionary without a full
rewrite. I left that out on purpose because proving it needs a full
attribute scan, but a dedicated "verify and drop" function the way you
describe is a reasonable way to expose it, and I'd take it as a follow-up.

So, concretely, where I'd land:

- The two efforts aren't competitors; this one is the TOAST-specific
consumer, yours is the general facility. If pg_encoding_context lands,
the TOAST dictionary catalog is a natural thing to re-express on top of
it, with the column-local lifecycle kept as a TOAST-side policy.
- I'd rather not block this series on that re-expression. The containment
and retirement rules are the reviewed-and-tested core here, and they're
TOAST-shaped; rebasing them onto a generic store is a mechanical change
that can happen once the generic store exists and is agreed on.
- I'm happy to review the generic patch with the TOAST case as a concrete
first consumer, and to see whether the multiple-dictionary and
verify-and-drop pieces are better in the common layer than in TOAST.

Does that division make sense to you, or do you see the containment rule
fitting into the generic context after all? That's the piece I can't yet
see clearly.

best.

-greg