| From: | Greg Burd <greg(at)burd(dot)me> |
|---|---|
| To: | Hannu Krosing <hannuk(at)google(dot)com> |
| Cc: | pgsql-hackers(at)lists(dot)postgresql(dot)org, Manu <manuelreyesbravo(at)gmail(dot)com>, Nikhil Kumar Veldanda <veldanda(dot)nikhilkumar17(at)gmail(dot)com>, Michael Paquier <michael(at)paquier(dot)xyz>, Japin Li <japinli(at)hotmail(dot)com>, Dharin Shah <dharinshah95(at)gmail(dot)com> |
| Subject: | Re: ZSTD TOAST compression, and an extensible compression method encoding |
| Date: | 2026-11-20 15:13:39 |
| Message-ID: | 179517866494.3567880.17795393185833942329@burd.me |
| Views: | Whole Thread | Raw Message | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-hackers |
On Thursday, November 19th, 2026 at 3:27 PM, Hannu Krosing <hannuk(at)google(dot)com> wrote:
> Hi Greg,
Hi Hannu,
Thanks for the careful pass over v19, and for separating the one real
bug from the polish.
> If there is any chance 0004 lands ahead of 0005, I'd vote for moving the
> GUC into 0004 since 0004 is what opens pg_train_toast_dictionary() to
> MAINTAIN. (Naming it `toast_dictionary_max_retained` might also read a
> bit more naturally now that it applies to manual training even when
> `dictionary_evolution = off`, though keeping the current name if 0004 and
> 0005 go in together is fine too.)
I left the GUC in 0005 for v20. The cap only has teeth once 0005 adds
the retention logic: in 0004 alone, pg_train_toast_dictionary() keeps
every dictionary it trains and a VACUUM FULL is the only way to shed
them, so a bound there would have nothing to enforce. 0004 and 0005 are
meant to go in together, and the name reads fine that way. If a
committer wants to take 0004 on its own, I'll move the GUC and the two
cap checks into it and rename it toast_dictionary_max_retained in the
same motion; it's a mechanical split and I'd rather do it when we know
0004 is landing alone than carry a cap that does nothing in the
single-patch case.
> 1. Unconditional tdtrainrows reset on non-concurrent REFRESH MATERIALIZED
> VIEW prevents matviews from ever evolving (matview.c, 0005)
[...]
> if (skipData || processed == 0)
> toast_dictionary_retire(matviewOid, true);
>
> resets the baseline whenever a refresh empties the matview while letting
> populated refreshes evolve normally across ANALYZE runs.
Fixed as you suggested. A populated REFRESH now keeps the recorded row
count so evolution can progress across refresh cycles; a refresh that
empties the matview (WITH NO DATA, or a query that returns no rows)
still resets the baseline the way TRUNCATE does. The regression test in
0005 now checks both arms: it trains a dictionary, does a populated
REFRESH and confirms the baseline is kept, then a REFRESH ... WITH NO
DATA and confirms it drops back to zero.
> 2. Three md5() calls left in 0004's SQL tests (FIPS-mode OpenSSL)
Switched all three to fipshash(), matching compression_zstd.sql.
> - verify_heapam.c:1680 (0002 & 0004): the new chunk-0 header check is
> gated by `chunksize >= (int32) sizeof(uint32)`, so if chunk 0 of a
> compressed external value is truncated below 4 bytes (0002), or has
> 5..8 bytes on a zstd_dict / lz4_dict value (truncating the 4-byte
> dictid, 0004), `verify_heapam(..., check_toast => true)` reports 0
> errors while detoasting fails with `compressed [dictionary] data is
> corrupt`.
Fixed. The check no longer skips a too-short chunk 0; it reports the
truncation. 0002 reports a chunk 0 shorter than the 4-byte va_tcinfo,
and, once the method is known, shorter than the header that method needs
(5 bytes for the long form's method byte). 0004 extends that to the
dictionary methods, whose header carries a 4-byte dictionary OID after
the method byte (VARHDRSZ_COMPRESSED_DICT). So a value whose chunk 0
stops inside the header is now a reported corruption rather than a clean
verify_heapam() followed by a detoast failure.
> - 007_verify_toast_pointer.pl:55 (0002): comment still says `one byte
> 0x01` three lines below `$va_1b_e`.
Fixed. The comment now points at $va_1b_e rather than naming the
little-endian byte.
> - pg_dump TAP tests (0004 & 0005): 0003 added `test_compression_zstd` to
> 006_pg_dump_compress.pl; adding `zstd_dict`/`lz4_dict` cases there and
> a `dictionary_evolution` attoptions check in 002_pg_dump.pl would round
> out the pg_dump test coverage (the C code itself dumps both fine).
Added to 006_pg_dump_compress.pl: a zstd_dict case and an lz4_dict case
alongside 0003's zstd case, each checking the dumped SET COMPRESSION
line, and a third case that sets dictionary_evolution and checks the
dumped attoption. I put the attoption check in 006 rather than
002_pg_dump.pl: 002 creates its tables unconditionally, so a zstd_dict
column there would break the suite on a build without zstd, whereas 006
already gates each case on compile_option. The lz4_dict case gates on
lz4 alone, since CREATE TABLE ... COMPRESSION lz4_dict needs only liblz4
(the zstd trainer is a train-time dependency, not a DDL-time one).
> - SGML nits (0004):
Both fixed. create_table.sgml and alter_table.sgml now list zstd_dict
and lz4_dict in the supported-methods sentence (pointing at the
paragraphs below), and the stray and missing blank lines in
system-views.sgml are gone.
v20 attached, rebased onto 0cb41ee8070. Changes since v19:
0002: verify_heapam() reports a chunk 0 truncated short of the
compressed header instead of passing it; fixed the stale $va_1b_e
comment in 007_verify_toast_pointer.pl.
0004: verify_heapam() extends that check to the dictionary methods'
4-byte dictid; three md5() calls in the SQL tests become
fipshash(); added zstd_dict and lz4_dict pg_dump cases;
create_table/alter_table list both dictionary methods; fixed the
system-views.sgml blank lines.
0005: a populated REFRESH MATERIALIZED VIEW keeps the evolution
baseline, only an empty refresh resets it, with the test covering
both; added the dictionary_evolution attoption pg_dump case.
0001-0003 are unchanged but for the two 0002 fixes above. With the
matview bug fixed I think 0004-0005 are ready too. I'd be happy to hear
whether a committer will pick the series up.
-greg
| Attachment | Content-Type | Size |
|---|---|---|
| v20-0001-Decode-TOAST-pointers-fully-in-toast_external_in.patch | text/x-patch | 13.0 KB |
| v20-0002-Allow-more-than-four-TOAST-compression-methods.patch | text/x-patch | 52.8 KB |
| v20-0005-Add-automatic-dictionary-evolution-during-ANALYZ.patch | text/x-patch | 103.0 KB |
| v20-0003-Add-zstd-as-a-TOAST-compression-method.patch | text/x-patch | 97.1 KB |
| v20-0004-Add-TOAST-compression-dictionaries-for-zstd-and-.patch | text/x-patch | 181.6 KB |
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Peter Eisentraut | 2026-11-20 15:17:27 | Re: compiler attribute alloc_size |
| Previous Message | Heikki Linnakangas | 2026-11-20 15:11:35 | Re: Assertion with aborted UPDATE in subtransaction |