Re: ZSTD TOAST compression, and an extensible compression method encoding

From: Hannu Krosing <hannuk(at)google(dot)com>
To: Greg Burd <greg(at)burd(dot)me>
Cc: pgsql-hackers(at)lists(dot)postgresql(dot)org, Manu <manuelreyesbravo(at)gmail(dot)com>, Nikhil Kumar Veldanda <veldanda(dot)nikhilkumar17(at)gmail(dot)com>, Michael Paquier <michael(at)paquier(dot)xyz>, Japin Li <japinli(at)hotmail(dot)com>, Dharin Shah <dharinshah95(at)gmail(dot)com>
Subject: Re: ZSTD TOAST compression, and an extensible compression method encoding
Date: 2026-11-19 20:26:52
Message-ID: CAMT0RQQmUM0nnw4cso34F=F__c6zXJRyCkDLc1uTm+UN8yr-NQ@mail.gmail.com
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hi Greg,

On Tue, Nov 17, 2026 at 6:54 PM Greg Burd <greg(at)burd(dot)me> wrote:
> Thank you both. v19 is attached; it applies on master as of
> 11591f273e6.

Thanks for v19, and especially for attaching notes.txt -- having the exact
H*/M*/R* mapping and test names made this pass a pleasure to verify.

I ran v19 on current master under ASan + UBSan + cassert, pgindent, the
pg_upgrade suites (including matviews and dropped zstd_dict columns),
SP-GiST / GIN / BRIN index checks, and a concurrent UPDATE loop
overlapping REPACK (CONCURRENTLY).

All of the v17 items (and your R* fixes, including the liblz4 < 1.9.0
stream fix and the cap minimum/baseline fixes) look good. 0001-0003 look
Ready for Committer to me, and 0004-0005 only have one real logic bug in
v19's matview refresh change plus a FIPS test nit and tiny polish.

=== On moving max_retained into 0004 ===

> On the cap, I made max_retained bound manual training too, for everyone
> but superusers. That seemed better than going back to owner-only: the
> MAINTAIN role that could loop the function can also VACUUM FULL, so it
> can clear the cap the normal way, and owners get the same limit, which
> is easier to explain. The cap lives in 0005, so 0004 on its own still
> has no limit. I'm on the fence with regards to moving the GUC into
> 0004.

Your rule (bounding non-superuser manual training too, bumping min to 2,
and restarting only the cooldown when hitting the cap) makes a lot of
sense.

If there is any chance 0004 lands ahead of 0005, I'd vote for moving the
GUC into 0004 since 0004 is what opens pg_train_toast_dictionary() to
MAINTAIN. (Naming it `toast_dictionary_max_retained` might also read a
bit more naturally now that it applies to manual training even when
`dictionary_evolution = off`, though keeping the current name if 0004 and
0005 go in together is fine too.)

=== Remaining items in v19 ===

1. Unconditional tdtrainrows reset on non-concurrent REFRESH MATERIALIZED
VIEW prevents matviews from ever evolving (matview.c, 0005)

In RefreshMatViewByOid() (matview.c:354):

refresh_by_heap_swap(matviewOid, OIDNewHeap, relpersistence);
toast_dictionary_retire(matviewOid, true);

For `REFRESH MATERIALIZED VIEW mv WITH NO DATA` (or a refresh that
populates 0 rows), resetting `tdtrainrows = 0` like TRUNCATE is right.

However, doing it unconditionally on every populated non-concurrent
`REFRESH MATERIALIZED VIEW mv` breaks automatic evolution on matviews:
unlike a regular table (which grows via INSERTs after TRUNCATE + ANALYZE
sets a baseline), a matview can *only* grow across REFRESH cycles. With
`tdtrainrows` reset to 0 on every plain REFRESH, the next `ANALYZE mv`
always enters the "row count unknown" branch in
toast_dictionary_maybe_evolve():

if (OidIsValid(curid) && currows <= 0)
{
if (totalrows > 0)
dict_set_gates(onerel, attnum, (int64) totalrows, curtime);
goto done;
}

so it records the new row count and jumps to `goto done` without ever
testing a candidate dictionary (e.g., 300 rows -> ANALYZE installs dict1
-> plain REFRESH to 1,500 rows -> ANALYZE just sets tdtrainrows=1500 ->
plain REFRESH to 6,000 rows -> ANALYZE just sets tdtrainrows=6000, keeping
dict1 forever, whereas REFRESH CONCURRENTLY evolves to dict2).

Since RefreshMatViewByOid() already tracks `uint64 processed`, and
finish_heap_swap() inside refresh_by_heap_swap() has already called
`toast_dictionary_retire(OIDOldHeap, false)` to delete superseded
dictionaries, guarding the reset with:

if (skipData || processed == 0)
toast_dictionary_retire(matviewOid, true);

resets the baseline whenever a refresh empties the matview while letting
populated refreshes evolve normally across ANALYZE runs.

2. Three md5() calls left in 0004's SQL tests (FIPS-mode OpenSSL)

In R2 you switched t/007 from md5() to sha256() for FIPS builds, but
three md5() calls remain in 0004's SQL regression tests:
- compression_zstd_dict.sql: doc(i) (`md5(i::text)`) and cmwide
(`repeat(md5('wide'), 8000)`)
- compression_lz4_dict.sql: doc(i) (`md5(i::text)`)

Switching those three to `fipshash(...)` (like compression_zstd.sql)
avoids failures on FIPS-enabled OpenSSL animals.

=== Minor polish ===

- verify_heapam.c:1680 (0002 & 0004): the new chunk-0 header check is
gated by `chunksize >= (int32) sizeof(uint32)`, so if chunk 0 of a
compressed external value is truncated below 4 bytes (0002), or has
5..8 bytes on a zstd_dict / lz4_dict value (truncating the 4-byte
dictid, 0004), `verify_heapam(..., check_toast => true)` reports 0
errors while detoasting fails with `compressed [dictionary] data is
corrupt`.
- 007_verify_toast_pointer.pl:55 (0002): comment still says `one byte
0x01` three lines below `$va_1b_e`.
- pg_dump TAP tests (0004 & 0005): 0003 added `test_compression_zstd` to
006_pg_dump_compress.pl; adding `zstd_dict`/`lz4_dict` cases there and
a `dictionary_evolution` attoptions check in 002_pg_dump.pl would round
out the pg_dump test coverage (the C code itself dumps both fine).
- SGML nits (0004):
* create_table.sgml:372 and alter_table.sgml:471 still say "The
supported compression methods are pglz, lz4, and zstd." right before
the new paragraphs introducing zstd_dict and lz4_dict.
* system-views.sgml: stray extra blank line added between pg_tables and
pg_timezone_abbrevs, and missing blank line before
pg_toast_dictionaries.

Cheers,
Hannu

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Peter Geoghegan 2026-11-19 21:10:48 Re: [PATCH v2] Fix SP-GiST index-only scans that return only INCLUDE columns
Previous Message Andres Freund 2026-11-19 20:01:34 Re: Hash aggregate collisions cause excessive spilling