Re: ZSTD TOAST compression, and an extensible compression method encoding

From: Greg Burd <greg(at)burd(dot)me>
To: Hannu Krosing <hannuk(at)google(dot)com>
Cc: pgsql-hackers(at)lists(dot)postgresql(dot)org, Manu <manuelreyesbravo(at)gmail(dot)com>, Nikhil Kumar Veldanda <veldanda(dot)nikhilkumar17(at)gmail(dot)com>, Michael Paquier <michael(at)paquier(dot)xyz>, Japin Li <japinli(at)hotmail(dot)com>, Dharin Shah <dharinshah95(at)gmail(dot)com>
Subject: Re: ZSTD TOAST compression, and an extensible compression method encoding
Date: 2026-11-20 15:13:39
Message-ID: 179517866494.3567880.17795393185833942329@burd.me
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers


On Thursday, November 19th, 2026 at 3:27 PM, Hannu Krosing <hannuk(at)google(dot)com> wrote:

> Hi Greg,

Hi Hannu,

Thanks for the careful pass over v19, and for separating the one real
bug from the polish.

> If there is any chance 0004 lands ahead of 0005, I'd vote for moving the
> GUC into 0004 since 0004 is what opens pg_train_toast_dictionary() to
> MAINTAIN. (Naming it `toast_dictionary_max_retained` might also read a
> bit more naturally now that it applies to manual training even when
> `dictionary_evolution = off`, though keeping the current name if 0004 and
> 0005 go in together is fine too.)

I left the GUC in 0005 for v20. The cap only has teeth once 0005 adds
the retention logic: in 0004 alone, pg_train_toast_dictionary() keeps
every dictionary it trains and a VACUUM FULL is the only way to shed
them, so a bound there would have nothing to enforce. 0004 and 0005 are
meant to go in together, and the name reads fine that way. If a
committer wants to take 0004 on its own, I'll move the GUC and the two
cap checks into it and rename it toast_dictionary_max_retained in the
same motion; it's a mechanical split and I'd rather do it when we know
0004 is landing alone than carry a cap that does nothing in the
single-patch case.

> 1. Unconditional tdtrainrows reset on non-concurrent REFRESH MATERIALIZED
> VIEW prevents matviews from ever evolving (matview.c, 0005)
[...]
> if (skipData || processed == 0)
> toast_dictionary_retire(matviewOid, true);
>
> resets the baseline whenever a refresh empties the matview while letting
> populated refreshes evolve normally across ANALYZE runs.

Fixed as you suggested. A populated REFRESH now keeps the recorded row
count so evolution can progress across refresh cycles; a refresh that
empties the matview (WITH NO DATA, or a query that returns no rows)
still resets the baseline the way TRUNCATE does. The regression test in
0005 now checks both arms: it trains a dictionary, does a populated
REFRESH and confirms the baseline is kept, then a REFRESH ... WITH NO
DATA and confirms it drops back to zero.

> 2. Three md5() calls left in 0004's SQL tests (FIPS-mode OpenSSL)

Switched all three to fipshash(), matching compression_zstd.sql.

> - verify_heapam.c:1680 (0002 & 0004): the new chunk-0 header check is
> gated by `chunksize >= (int32) sizeof(uint32)`, so if chunk 0 of a
> compressed external value is truncated below 4 bytes (0002), or has
> 5..8 bytes on a zstd_dict / lz4_dict value (truncating the 4-byte
> dictid, 0004), `verify_heapam(..., check_toast => true)` reports 0
> errors while detoasting fails with `compressed [dictionary] data is
> corrupt`.

Fixed. The check no longer skips a too-short chunk 0; it reports the
truncation. 0002 reports a chunk 0 shorter than the 4-byte va_tcinfo,
and, once the method is known, shorter than the header that method needs
(5 bytes for the long form's method byte). 0004 extends that to the
dictionary methods, whose header carries a 4-byte dictionary OID after
the method byte (VARHDRSZ_COMPRESSED_DICT). So a value whose chunk 0
stops inside the header is now a reported corruption rather than a clean
verify_heapam() followed by a detoast failure.

> - 007_verify_toast_pointer.pl:55 (0002): comment still says `one byte
> 0x01` three lines below `$va_1b_e`.

Fixed. The comment now points at $va_1b_e rather than naming the
little-endian byte.

> - pg_dump TAP tests (0004 & 0005): 0003 added `test_compression_zstd` to
> 006_pg_dump_compress.pl; adding `zstd_dict`/`lz4_dict` cases there and
> a `dictionary_evolution` attoptions check in 002_pg_dump.pl would round
> out the pg_dump test coverage (the C code itself dumps both fine).

Added to 006_pg_dump_compress.pl: a zstd_dict case and an lz4_dict case
alongside 0003's zstd case, each checking the dumped SET COMPRESSION
line, and a third case that sets dictionary_evolution and checks the
dumped attoption. I put the attoption check in 006 rather than
002_pg_dump.pl: 002 creates its tables unconditionally, so a zstd_dict
column there would break the suite on a build without zstd, whereas 006
already gates each case on compile_option. The lz4_dict case gates on
lz4 alone, since CREATE TABLE ... COMPRESSION lz4_dict needs only liblz4
(the zstd trainer is a train-time dependency, not a DDL-time one).

> - SGML nits (0004):

Both fixed. create_table.sgml and alter_table.sgml now list zstd_dict
and lz4_dict in the supported-methods sentence (pointing at the
paragraphs below), and the stray and missing blank lines in
system-views.sgml are gone.

v20 attached, rebased onto 0cb41ee8070. Changes since v19:

0002: verify_heapam() reports a chunk 0 truncated short of the
compressed header instead of passing it; fixed the stale $va_1b_e
comment in 007_verify_toast_pointer.pl.
0004: verify_heapam() extends that check to the dictionary methods'
4-byte dictid; three md5() calls in the SQL tests become
fipshash(); added zstd_dict and lz4_dict pg_dump cases;
create_table/alter_table list both dictionary methods; fixed the
system-views.sgml blank lines.
0005: a populated REFRESH MATERIALIZED VIEW keeps the evolution
baseline, only an empty refresh resets it, with the test covering
both; added the dictionary_evolution attoption pg_dump case.

0001-0003 are unchanged but for the two 0002 fixes above. With the
matview bug fixed I think 0004-0005 are ready too. I'd be happy to hear
whether a committer will pick the series up.

-greg

Attachment Content-Type Size
v20-0001-Decode-TOAST-pointers-fully-in-toast_external_in.patch text/x-patch 13.0 KB
v20-0002-Allow-more-than-four-TOAST-compression-methods.patch text/x-patch 52.8 KB
v20-0005-Add-automatic-dictionary-evolution-during-ANALYZ.patch text/x-patch 103.0 KB
v20-0003-Add-zstd-as-a-TOAST-compression-method.patch text/x-patch 97.1 KB
v20-0004-Add-TOAST-compression-dictionaries-for-zstd-and-.patch text/x-patch 181.6 KB

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Peter Eisentraut 2026-11-20 15:17:27 Re: compiler attribute alloc_size
Previous Message Heikki Linnakangas 2026-11-20 15:11:35 Re: Assertion with aborted UPDATE in subtransaction