| From: | Hannu Krosing <hannuk(at)google(dot)com> |
|---|---|
| To: | Greg Burd <greg(at)burd(dot)me> |
| Cc: | pgsql-hackers(at)lists(dot)postgresql(dot)org, Manu <manuelreyesbravo(at)gmail(dot)com>, Nikhil Kumar Veldanda <veldanda(dot)nikhilkumar17(at)gmail(dot)com>, Michael Paquier <michael(at)paquier(dot)xyz>, Japin Li <japinli(at)hotmail(dot)com>, Dharin Shah <dharinshah95(at)gmail(dot)com> |
| Subject: | Re: ZSTD TOAST compression, and an extensible compression method encoding |
| Date: | 2026-11-19 20:26:52 |
| Message-ID: | CAMT0RQQmUM0nnw4cso34F=F__c6zXJRyCkDLc1uTm+UN8yr-NQ@mail.gmail.com |
| Views: | Whole Thread | Raw Message | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-hackers |
Hi Greg,
On Tue, Nov 17, 2026 at 6:54 PM Greg Burd <greg(at)burd(dot)me> wrote:
> Thank you both. v19 is attached; it applies on master as of
> 11591f273e6.
Thanks for v19, and especially for attaching notes.txt -- having the exact
H*/M*/R* mapping and test names made this pass a pleasure to verify.
I ran v19 on current master under ASan + UBSan + cassert, pgindent, the
pg_upgrade suites (including matviews and dropped zstd_dict columns),
SP-GiST / GIN / BRIN index checks, and a concurrent UPDATE loop
overlapping REPACK (CONCURRENTLY).
All of the v17 items (and your R* fixes, including the liblz4 < 1.9.0
stream fix and the cap minimum/baseline fixes) look good. 0001-0003 look
Ready for Committer to me, and 0004-0005 only have one real logic bug in
v19's matview refresh change plus a FIPS test nit and tiny polish.
=== On moving max_retained into 0004 ===
> On the cap, I made max_retained bound manual training too, for everyone
> but superusers. That seemed better than going back to owner-only: the
> MAINTAIN role that could loop the function can also VACUUM FULL, so it
> can clear the cap the normal way, and owners get the same limit, which
> is easier to explain. The cap lives in 0005, so 0004 on its own still
> has no limit. I'm on the fence with regards to moving the GUC into
> 0004.
Your rule (bounding non-superuser manual training too, bumping min to 2,
and restarting only the cooldown when hitting the cap) makes a lot of
sense.
If there is any chance 0004 lands ahead of 0005, I'd vote for moving the
GUC into 0004 since 0004 is what opens pg_train_toast_dictionary() to
MAINTAIN. (Naming it `toast_dictionary_max_retained` might also read a
bit more naturally now that it applies to manual training even when
`dictionary_evolution = off`, though keeping the current name if 0004 and
0005 go in together is fine too.)
=== Remaining items in v19 ===
1. Unconditional tdtrainrows reset on non-concurrent REFRESH MATERIALIZED
VIEW prevents matviews from ever evolving (matview.c, 0005)
In RefreshMatViewByOid() (matview.c:354):
refresh_by_heap_swap(matviewOid, OIDNewHeap, relpersistence);
toast_dictionary_retire(matviewOid, true);
For `REFRESH MATERIALIZED VIEW mv WITH NO DATA` (or a refresh that
populates 0 rows), resetting `tdtrainrows = 0` like TRUNCATE is right.
However, doing it unconditionally on every populated non-concurrent
`REFRESH MATERIALIZED VIEW mv` breaks automatic evolution on matviews:
unlike a regular table (which grows via INSERTs after TRUNCATE + ANALYZE
sets a baseline), a matview can *only* grow across REFRESH cycles. With
`tdtrainrows` reset to 0 on every plain REFRESH, the next `ANALYZE mv`
always enters the "row count unknown" branch in
toast_dictionary_maybe_evolve():
if (OidIsValid(curid) && currows <= 0)
{
if (totalrows > 0)
dict_set_gates(onerel, attnum, (int64) totalrows, curtime);
goto done;
}
so it records the new row count and jumps to `goto done` without ever
testing a candidate dictionary (e.g., 300 rows -> ANALYZE installs dict1
-> plain REFRESH to 1,500 rows -> ANALYZE just sets tdtrainrows=1500 ->
plain REFRESH to 6,000 rows -> ANALYZE just sets tdtrainrows=6000, keeping
dict1 forever, whereas REFRESH CONCURRENTLY evolves to dict2).
Since RefreshMatViewByOid() already tracks `uint64 processed`, and
finish_heap_swap() inside refresh_by_heap_swap() has already called
`toast_dictionary_retire(OIDOldHeap, false)` to delete superseded
dictionaries, guarding the reset with:
if (skipData || processed == 0)
toast_dictionary_retire(matviewOid, true);
resets the baseline whenever a refresh empties the matview while letting
populated refreshes evolve normally across ANALYZE runs.
2. Three md5() calls left in 0004's SQL tests (FIPS-mode OpenSSL)
In R2 you switched t/007 from md5() to sha256() for FIPS builds, but
three md5() calls remain in 0004's SQL regression tests:
- compression_zstd_dict.sql: doc(i) (`md5(i::text)`) and cmwide
(`repeat(md5('wide'), 8000)`)
- compression_lz4_dict.sql: doc(i) (`md5(i::text)`)
Switching those three to `fipshash(...)` (like compression_zstd.sql)
avoids failures on FIPS-enabled OpenSSL animals.
=== Minor polish ===
- verify_heapam.c:1680 (0002 & 0004): the new chunk-0 header check is
gated by `chunksize >= (int32) sizeof(uint32)`, so if chunk 0 of a
compressed external value is truncated below 4 bytes (0002), or has
5..8 bytes on a zstd_dict / lz4_dict value (truncating the 4-byte
dictid, 0004), `verify_heapam(..., check_toast => true)` reports 0
errors while detoasting fails with `compressed [dictionary] data is
corrupt`.
- 007_verify_toast_pointer.pl:55 (0002): comment still says `one byte
0x01` three lines below `$va_1b_e`.
- pg_dump TAP tests (0004 & 0005): 0003 added `test_compression_zstd` to
006_pg_dump_compress.pl; adding `zstd_dict`/`lz4_dict` cases there and
a `dictionary_evolution` attoptions check in 002_pg_dump.pl would round
out the pg_dump test coverage (the C code itself dumps both fine).
- SGML nits (0004):
* create_table.sgml:372 and alter_table.sgml:471 still say "The
supported compression methods are pglz, lz4, and zstd." right before
the new paragraphs introducing zstd_dict and lz4_dict.
* system-views.sgml: stray extra blank line added between pg_tables and
pg_timezone_abbrevs, and missing blank line before
pg_toast_dictionaries.
Cheers,
Hannu
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Peter Geoghegan | 2026-11-19 21:10:48 | Re: [PATCH v2] Fix SP-GiST index-only scans that return only INCLUDE columns |
| Previous Message | Andres Freund | 2026-11-19 20:01:34 | Re: Hash aggregate collisions cause excessive spilling |