Re: ZSTD TOAST compression, and an extensible compression method encoding

From: Greg Burd <greg(at)burd(dot)me>
To: Hannu Krosing <hannuk(at)google(dot)com>
Cc: pgsql-hackers(at)lists(dot)postgresql(dot)org, Manu <manuelreyesbravo(at)gmail(dot)com>, Nikhil Kumar Veldanda <veldanda(dot)nikhilkumar17(at)gmail(dot)com>, Michael Paquier <michael(at)paquier(dot)xyz>, Japin Li <japinli(at)hotmail(dot)com>, Dharin Shah <dharinshah95(at)gmail(dot)com>
Subject: Re: ZSTD TOAST compression, and an extensible compression method encoding
Date: 2026-11-17 17:54:52
Message-ID: avyT4tQ2fzTYdnsu@werk
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On Sun, Nov 15, 2026 at 11:33 UTC, Hannu Krosing <hannuk(at)google(dot)com> wrote:
> I am still working on a (more widely ambitious :) ) Direct TOAST project
> some parts of which also have the dictionary-based compression
> implementation, so following this has been very informative.
>
> This also makes me interested to see this one moving forward in a way
> that will not blow up maintainer load in the future.

Hi Hannu,

Same goal here. If the dictionary lifecycle here turns out to be
something Direct TOAST can reuse, even better.

> 1. Big-endian failure in contrib/amcheck/t/007_verify_toast_pointer.pl

Good catch, and embarrassing: I checked the endianness for va_extinfo
and then hardcoded the header byte two lines above. The test now
expects 0x80 or 0x01 depending on the same pack() test. I have no
big-endian box to run it on, so that one is by reading varatt.h only.

> 2. RelationInvalidateRelation() leaves a rolled-back subtransaction's
> dictionary OID cached in rd_toastdictoids (relcache.c, 0004)

Reproduced it with your trigger exactly, and fixed it the way you
suggest: RelationInvalidateRelation() now frees rd_toastdictoids and
rd_toastdictvalid next to rd_amcache. Your case is in
compression_zstd_dict now; it fails with "compression dictionary N does
not exist" without the fix.

> 3. pg_train_toast_dictionary() ACL_MAINTAIN nuances (0004 & 0005)

All three. The function now accepts what ANALYZE accepts (MAINTAIN, or
owning the database), checks the relation kind before taking the lock,
and checks the privilege again once it holds the lock.

On the cap, I made max_retained bound manual training too, for everyone
but superusers. That seemed better than going back to owner-only: the
MAINTAIN role that could loop the function can also VACUUM FULL, so it
can clear the cap the normal way, and owners get the same limit, which
is easier to explain. The cap lives in 0005, so 0004 on its own still
has no limit. I'm on the fence with regards to moving the GUC into
0004.

While there I found two problems with the cap itself. It counts the
current dictionary, so at its old minimum of 1 a column could never be
trained again and the hint about VACUUM FULL could never help; the
minimum is 2 now. And when evolution hit the cap it restarted both
gates, so after the rewrite it waited for the table to double again
before retraining. It now restarts only the cooldown, and a test covers
resuming after a rewrite.

> 4. Sample starvation in pg_train_toast_dictionary() when dict_size <= 9175
> on wide rows or early outliers (0004)

Yes, and it is worse than an edge case: a 4kB dictionary on a column of
128kB values could never be trained. I used your cap on the per-value
slice, a tenth of the budget. Both of your examples train now, and
there's a test with one wide value in front of normal ones.

> 5. Missing DECLARE_FOREIGN_KEY in pg_toast_dictionary.h (0004)

Added, and oidjoins checks it.

> - 0002: verify_heapam() flags `!is_compressed && VARTAG_IS_ONDISK_LONG(tag)`
> ("is not compressed but has a long TOAST pointer"), whereas
> toast_fetch_datum[_slice]() and toast_get_compression_id() accept an
> uncompressed long pointer silently

Agreed; the readers refuse it now too, through one function that
toast_get_compression_id() also calls. It's worse than a mismatch: a
reader would have returned the compressed bytes as the value. t/007
forges one by shrinking va_rawsize.

> - 0004: dict_get_ddict() creates dict_dctx without setting
> ZSTD_d_windowLogMax

Fixed; the limit moved into toast_compression.h so both contexts use it.
(Strictly it only matters for streaming decompression, which the
dictionary path doesn't use yet, so the comment now says so.)

> compression_zstd_dict.sql comments "-- values nested in containers:
> rows, arrays, ranges of text are flattened", though cmnest only tests
> a composite and an array

The comment promised more than the test checked. It has a textrange
now.

> - 0005: `REFRESH MATERIALIZED VIEW mv WITH NO DATA` ... leaving the
> pre-empty tdtrainrows

Fixed for any non-concurrent REFRESH, which replaces the contents just
like TRUNCATE. The test no longer pokes the catalog to set it up.

> zstd_dict_compressed_total() still has bare
> `out = palloc(ZSTD_compressBound(maxlen));`

Fixed, and lz4_dict_compressed_total() too.

> Under `--without-zstd --with-lz4`, an `#ifndef USE_ZSTD return;
> #endif` at step (a) of toast_dictionary_maybe_evolve()

Done.

On Sun, Nov 15, 2026 at 04:53 UTC, Manu <manuelreyesbravo(at)gmail(dot)com> wrote:
> From 0004 on, with the last ID at 5, that check alone keeps a long
> pointer with byte 3 away from the Assert in
> toast_compression_id_needs_cmid_byte(), and t/007 has no byte-3 case.

Thanks, that is exactly the kind of hole I wanted t/007 to close.
There's a byte-3 row now.

Your other remark in checks.txt, that a zstd value whose pointer says 4
or 5 goes unnoticed because readers trust the stored header, also seemed
worth fixing. amcheck now compares the pointer's method with the
compressed header at the start of the first chunk (with check_toast).
An invalid header there is reported by what is stored, like the pointer
checks, and a valid but different method by both IDs. t/007 has the
second case from 0004 on, where 4 is a real method; in 0003 the same
forged byte is just an unknown method. The scripts were a big help, I
reused their layout for the new cases.

What else changed:

- t/007 read the forged uncompressed value with length(), which
answers from va_rawsize without fetching anything, so the test
passed or failed depending on the platform. CI caught it on 32-bit,
autoconf and MinGW. It reads the value now. The test also used
md5(), which FIPS builds refuse, and its zstd values stayed inline
at 32kB blocks; both fixed.

- lz4_dict used LZ4_initStream() and LZ4_resetStream_fast(), which are
lz4 1.9.0 and later, while configure accepts any lz4. LZ4_loadDict()
resets the stream in every version, so the stream is now created
once with LZ4_createStream() and nothing else is needed.

- Smaller cleanups: the zstd_dict and lz4_dict cases in
toast_compress_datum_rel() are one case; the relkind check is done
once, with errdetail_relkind_not_supported(); DictCompressRoutine
loses train(), which both codecs pointed at the same function; the
docs say lz4 uses only the last 64kB of a dictionary, and the
superuser-only settings say so.

Thank you both. v19 is attached; it applies on master as of
11591f273e6.

best.

-greg

Attachment Content-Type Size
notes.txt text/plain 14.2 KB
v19-0002-Allow-more-than-four-TOAST-compression-methods.patch text/x-diff 51.6 KB
v19-0005-Add-automatic-dictionary-evolution-during-ANALYZ.patch text/x-diff 100.6 KB
v19-0003-Add-zstd-as-a-TOAST-compression-method.patch text/x-diff 97.1 KB
v19-0001-Decode-TOAST-pointers-fully-in-toast_external_in.patch text/x-diff 12.9 KB
v19-0004-Add-TOAST-compression-dictionaries-for-zstd-and-.patch text/x-diff 176.3 KB

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Maksim.Melnikov 2026-11-17 18:04:48 Re: Infinite loop in XLogPageRead() on standby
Previous Message Tomas Vondra 2026-11-17 17:22:51 Re: [PATCH]Fix planner's strictness check for JsonExpr