Re: ZSTD TOAST compression, and an extensible compression method encoding

Hi Greg,

> > Please take a look at a generalized design for the compression dictionary
> >
> > In email: [PATCH v1] Generic Datum Encoding and Compression Contexts
> > (pg_encoding_type, pg_encoding_context)
>
> Hey Hannu,
>
> Sure thing, I'll give that a look and see how the two designs might become
> one.

I have to check if that version already has "fallback encodings" to
try if the main one fails.

And one-byte in-group "variants" to have something small to store
inside the data itself as an indicator.

> > I have used this myself in dictionary compression for zstd and lz4 and
> > for keeping Apache Avro schemas for JSON and JSONB encodings and
> > defining cascaded Avro+zstd
>
> Interesting use cases.

Avro was especially efficient for JSONB because it didn't need to
parse anything; it could use the JSONB internal structure directly for
encoding.

And dictionary compression was in many cases faster than not using a
dictionary. So, dictionary compression is definitely worth doing.

> > And also I used the same for storing quantization tables for pgvector
> > int8 and other quantizations
>
> Hmmm... ok.
>
> > This could also be used for other "encoding" tasks, like having a
> > public key for writes and a more restricted private key for reads,
> > etc.
>
> Are we at risk of over-generalizing/loading this one idea too much? I'd
> imagine you to say "no", but I had to ask. :)

We might be in some cases, but then we find out and can either fall
back to case-specific tables, or add more fields and types to
pg_encoding_context

We do have about 10 very different object types in pg_class and
somehow survive :)

> > The main point is that having a generic facility for these kinds of
> > tasks would benefit many use cases.
>
> I'm all for reuse and a generic facility if one can be agreed upon.

Yup, would not be just saving on code but also on mental effort
learning many similar but subtly different ways of doing the same
thing :)

---
Hannu