Re: ZSTD TOAST compression, and an extensible compression method encoding

On Tue, Oct 27, 2026 at 12:53 PM Greg Burd <greg(at)burd(dot)me> wrote:
>
>
> On Monday, October 26th, 2026 at 3:02 PM, Greg Burd <greg(at)burd(dot)me> wrote:
> >
> > On Sunday, October 25th, 2026 at 12:52 PM, Hannu Krosing <hannuk(at)google(dot)com> wrote:
> >
> Hi Hannu,
>
> I read the Generic Datum Encoding and Compression Contexts patch. It is a
> bigger idea than this one, and worth pulling apart where the two meet and
> where they don't.

Thanks for a thoughtful review!

> Inline below.
>
> > Please take a look at a generalized design for the compression
> > dictionary [...] I have used this myself in dictionary compression for
> > zstd and lz4 and for keeping Apache Avro schemas for JSON and JSONB
> > encodings and defining cascaded Avro+zstd
>
> Where we agree: a trained dictionary is just one case of "a value needs a
> side artifact to be decoded." Your pg_encoding_context is that artifact
> in the general form, and a zstd/lz4 dictionary is one kind of content for
> it. The two designs even store nearly the same thing: an id, owning
> state, and a bytea blob.
>
> Where they differ is scope, and it is the scope that drove this design.

Yes, I left all the scope work to the user, so very little beyond
basig support for retiring is in the patch.

Because we lack reference-counted pg_depend (the possible need for
which was also revealed in recent PGQ retraction discussions) I
considered this something that should be tied to specific use, not the
context table itself.

> This series is about TOAST, specifically. The hard problem it solves is
> not "store a dictionary," it is "know when a dictionary can be dropped
> without a full table scan to prove no value still needs it."

In some of my designs I just llet this to the admin, so the sequence would be
- lock the context
- run migration off the context
- call drop_context_if_safe(ditionary, VARIADIC user_columns[])

that was done to allow using the same contexts across tables without
making the lifetime tracking frameworks too complex.

> The answer
> here is column-local ownership: a dictionary belongs to one (relid,
> attnum) like a pg_statistic row, values compressed with it never leave
> that column in compressed form (heap_fill_tuple marks them external so
> every self-contained copy detoasts them), and a rewrite is what retires
> the old one. That containment rule is the whole point, and it is TOAST
> machinery, not general datum machinery; I don't see how to express it in a
> facility that doesn't know about attcompression, detoast, and the rewrite
> path.

I think the high level lock-migrate-drop pattern still works, but the
details can be quite complex

> We'll have to consider this, and other use cases if your generalized
> method is going to work.
>
> > And also I used the same for storing quantization tables for pgvector
> > int8 and other quantizations [...] like having a public key for writes
> > and a more restricted private key for reads, etc. The main point is
> > that having a generic facility for these kinds of tasks would benefit
> > many use cases.
>
> This is the part I'd be careful with. Quantization tables, Avro schemas,
> and key management each have their own lifecycle and their own "when can
> this be dropped" answer, and those answers are not the same as a TOAST
> dictionary's. A facility general enough to hold all of them tends to push
> the lifecycle decision back onto each caller, which is exactly the part
> that was hard to get right here.

Yes, I started from the assumption that this belongs to the feature's user.

And I decided that until we have more capable dependency tracking,
I'll just fall back to manually managing this.

> I'm not against the generic facility; I
> am wary of making the TOAST feature wait on, or inherit the constraints
> of, a design that also has to satisfy pgvector and key rotation.

Agreed. For me this is agood use case to test the design :)

> > [from the follow-up] At least the underlying storage should allow
> > multiple dictionaries for the same relation and moving to newer
> > dictionaries or some heuristic for dictionary selection.
>
> That part we already do, and it's the strongest point of overlap.
> pg_toast_dictionary holds many dictionaries per column, each value carries
> the 4-byte OID of the one it was compressed with, exactly one is current
> (tdcurrent) and new writes use it, and 0005 adds the heuristic you
> describe: ANALYZE trains a candidate and adopts it only when a held-out
> test says it beats the incumbent. So the "multiple dictionaries, move to
> newer ones, select by heuristic" shape is in the series today.

Thanks for confirming also other see this popping up in multiple places.

> > It also would be nice to have an option for "retiring" a dictionary by
> > - setting it read-only, so no new rows can use it
> > - having some process re-encode old values (or [...] wait until
> > partitions with such values are dropped)
> > - then have a verify and drop function
>
> This is close to what happens, under different names. Marking a
> dictionary non-current is your "read-only": no new value uses it, old ones
> still read. The re-encode step is a table rewrite (VACUUM FULL, CLUSTER,
> REPACK), which recompresses every value with the current dictionary and
> then drops the non-current ones. What the series does not have, and your
> framing makes a fair case for, is a rewrite-free retirement: a verify-then-
> drop that proves no live value references a dictionary without a full
> rewrite. I left that out on purpose because proving it needs a full
> attribute scan, but a dedicated "verify and drop" function the way you
> describe is a reasonable way to expose it, and I'd take it as a follow-up.

And I was approaching the same issue from exactly the opposite
direction - considering the (VACUUM FULL, CLUSTER, REPACK) to be the
prohibitively expensive option for live databases :)

If there is a way to query for current dictionary one can do the
retiring of the locked ones as a lazy background process instead of
one-shot reorg.

> So, concretely, where I'd land:
>
> - The two efforts aren't competitors; this one is the TOAST-specific
> consumer, yours is the general facility. If pg_encoding_context lands,
> the TOAST dictionary catalog is a natural thing to re-express on top of
> it, with the column-local lifecycle kept as a TOAST-side policy.

Indeed. I will take a close look at the differences and similarities
of the dictionary stores and see how they would map to each other.

> - I'd rather not block this series on that re-expression. The containment
> and retirement rules are the reviewed-and-tested core here, and they're
> TOAST-shaped;

Yup, I sent a few more findings, but in general it looks good.

> rebasing them onto a generic store is a mechanical change
> that can happen once the generic store exists and is agreed on.
> - I'm happy to review the generic patch with the TOAST case as a concrete
> first consumer, and to see whether the multiple-dictionary and
> verify-and-drop pieces are better in the common layer than in TOAST.
>
> Does that division make sense to you, or do you see the containment rule
> fitting into the generic context after all? That's the piece I can't yet
> see clearly.

Absolutely makes sense. Thanks for taking the time to think about this :)

My main takeaway is that we have been spending our focus on different
parts of the same issue and, at least I hoe, there is a large overlap
that can be used at some point to reduce the complexity of similar
features,

--
Cheers
Hannu