Re: ZSTD TOAST compression, and an extensible compression method encoding

Hi everyone

I have been working on a more generic approach to compression and
similar encoding/decoding tasks

I have put my current thinking on this in
https://wiki.postgresql.org/wiki/Pg_encoding_context

This is meant to provide generic support functionality for things like
compression dictionaries for zstd and lz4, Avro schemas, huffman
tables and vector quanttization base data to name a few

I appreciate any feedback on the approach

---
Hannu

On Wed, Sep 30, 2026 at 8:20 AM Michael Paquier <michael(at)paquier(dot)xyz> wrote:
>
> On Mon, Sep 28, 2026 at 10:39:42PM -0700, Nikhil Kumar Veldanda wrote:
> > v7 attached
>
> I was just looking at v7-0001 that wants to add the ToastCompressionId
> to toast_external_data, and this feels half-baked due to the
> inconsistency this brings with extsize and VARATT_EXTINFO_GET_EXTSIZE.
>
> Couldn't we do better here by normalizing more data from the existing
> fields? Another could be the is_compressed state which is guessed
> from a comparison between the raw size and the compressed size,
> perhaps?
>
> Regarding v7-0002 and v7-0003, I am doubting the wisdom of tackling
> the last-compression-bit issue for this release. The OID8 code has
> already changed a lot of code, and maybe we should be conservative in
> terms of the amount of the changes we do in this area for a single
> release. By that, I mean to catch up on the refactoring pieces on
> this thread once some dust has settled on HEAD and tackle this issue
> around the time v21 opens up (Aka I'm preparing myself for these
> dozens of agents to complain about the shape of the code already
> committed, we'll see how it goes).
> --
> Michael