measurements-v11.txt -- ZSTD TOAST v11 review (uid 163493), 25 Oct 2026 Tree: master dfd7ede529 + v11-0001..0005 (git am clean). gcc 16.2.1, zstd 1.5.7, lz4 1.10.0. Scripts: tests-v11.txt. Counts and sizes only. 1. Builds and make check (build_v11.sh) build result both, -O2 0 warnings; check 243/243; amcheck ok both, cassert+ASan 0 warnings; check 243/243; amcheck ok; 0 ASan reports lz4 only FAILS: toast_dictionary.c:537 implicit declaration of dict_fetch zstd only builds; check 242/243: compression_zstd_dict "compression method lz4 not supported" neither 0 warnings; check 243/243 With nocfbot-v11-lz4-without-zstd.diff.txt: lz4 only and zstd only 0 warnings, check 243/243, amcheck ok. With both diffs: -O2 and cassert+ASan 0 warnings, check 243/243, amcheck ok, 0 ASan reports. 2. lz4-only build with the first diff (t_fix.sh): cluster written by a build with both libraries (both diffs), read by the lz4-only one: R1 803 lz4_dict values (3 external) read whole: true R2 slices: true; R3/R4 100 new lz4_dict rows written and read: true R5 never-trained lz4_dict column stored as lz4: true R6-R8 training, reading zstd_dict, new zstd_dict column: clean "requires ... zstd support" errors; R9 VACUUM FULL then read: true 3. Dictionary caches (t_cache.sh), 3000-byte json values, 64 kB dictionaries. fetch = dict_fetch() calls, ddict = ZSTD_createDDict() calls (gdb); instr = user-space instructions per value (perf stat, (6 queries - 1) / 5 / values). v11 with the cache diff case values fetch ddict instr fetch ddict instr zstd 1 dict 2000 1 1 23223 1 1 23223 zstd 2 dicts 4000 4000 4000 265236 2 2 27872 lz4 1 dict 2000 1 - 11501 1 - 11485 lz4 2 dicts 4000 4000 - 189002 2 - 10836 retrain, new 1000 1 1 23579 1 1 23599 retrain, mixed 2000 1946 1946 258163 2 2 27891 "retrain": one column, dictionary #1 rows, every other row deleted, VACUUM, retrained (#2), new rows into the freed slots: 1946 dictionary changes along the physical scan of 2000 rows. The two columns are separate runs of the same script; repeated runs move by tens of instructions per value. 4. Cache diff under invalidation and eviction (t_inval.sh, cassert+ASan): 10 of 10 true (another session retrains, VACUUM FULL, DROP COLUMN between reads of one backend; six dictionary columns; reads and writes with debug_discard_caches = 1); amcheck 0 corruption rows; 0 ASan. 5. Evolution in ANALYZE (t_evolve.sh), 60000 rows of 3000-byte json, one dictionary trained on the same distribution, no drift; every ANALYZE evaluates (growth 1.0, min_rows 0, min_interval 0), none adopts (min_gain 1.0); 12 ANALYZE e (v) per statistics target: target 10 (heldout=600) gain %: 0.9 0.9 1.4 1.5 1.6 1.7 1.7 1.9 1.9 2.0 2.0 2.3 target 100 (heldout=6000) gain %: 1.2 1.2 1.3 1.4 1.7 1.8 1.8 1.9 1.9 2.2 2.2 2.3 All 24 below the 5% default margin; 1 dictionary before and after. One evaluation at target 100: 12005 ZSTD_createCDict() calls (gdb). Cancel: a real SIGINT delivered on entry to toast_dictionary_maybe_evolve() (gdb, single-user backend): ANALYZE completes, the next statement runs, and the log has LOG "automatic dictionary evolution for column "v" of "e" was skipped: canceling statement due to user request". Control, SIGINT on entry to do_analyze_rel(): ERROR "canceling statement due to user request". 5b. ERROR inside the TOAST fetch of a sampled value (t_evolve_err.sh): one external zstd_dict value's chunk 1 renumbered to 77 on disk ("unexpected chunk number 77 (expected 1)", raised inside the TOAST index scan). ANALYZE, evolution off clean ANALYZE, evolution on returns; LOG "...was skipped: unexpected chunk number..."; 7 WARNING "resource was not closed": 3 buffer pins (toast table blk 7 x2, toast index blk 1), toast table, toast index x2, a TupleDesc. Same twice, same at -O2 (v11) and cassert+ASan (v11a); 0 ASan reports. 5c. With nocfbot-v11-evolve-subxact.diff.txt (internal subtransaction, re-throw on ERRCODE_QUERY_CANCELED or a pending interrupt), all four diffs, cassert+ASan: SIGINT on entry to toast_dictionary_maybe_evolve() ERROR "canceling statement due to user request" (as the control on do_analyze_rel()) same, mutant testing only QueryCancelPending/ProcDiePending WARNING "... was skipped: canceling statement ..." (swallowed: the flag is cleared before the ERROR is raised) damaged chunk (5b) 2 x WARNING "... was skipped: unexpected chunk number ...", 0 "resource was not closed", 0 ASan reports adoption (t_evolve_xact.sh, a new key distribution before each step) ANALYZE 1->2, BEGIN/ANALYZE/INSERT/COMMIT 2->3 (new rows zstd_dict), BEGIN/ANALYZE/ ROLLBACK 3, SAVEPOINT/ANALYZE/ROLLBACK TO 3, VACUUM (ANALYZE) 3->4; one current dictionary and all values correct after each; clean server log 5d. With nocfbot-v11-evolve-digest-once.diff.txt, one evaluation at 6000 held-out values, ZSTD_createCDict() by caller (gdb): before after held-out test 12000 2 ZDICT training (libzstd) 5 5 Same cluster, every row sampled (t_evolve_eq.sh, 20000 rows, zstd_dict and lz4_dict): the DEBUG1 evolution lines, cur= and cand= included, are identical before and after (4 of 4). 6. Corrupt values (t_corrupt.sh, cassert+ASan): 16 bytes of 0xff at offset 900 of chunk 1 of an external value. method full slice(1,100) subxact good values after lz4_dict error error error all true zstd_dict error error error all true lz4 error 100 bytes error all true 18 of 18 true, 0 false, 0 ASan reports. 7. Slices (t_slice.sh): substr() at 14 offsets x 4 lengths, 504 values (200 dictionary inline, 3 dictionary external ~300 kB, 301 compressed before training), against an uncompressed copy: 0 mismatches in each of 4 x 28224 checks (lz4_dict, zstd_dict; UTF8 with 2- and 3-byte characters; SQL_ASCII). 8. Ratio raw/stored (t_size.sh), 3000-byte values, 1000 training rows, 5000 evaluation rows (evaluation rows only): corpus pglz lz4 zstd lz4_dict 32k/64k/128k zstd_dict 32k/64k/128k docs 2.21 1.94 2.78 2.47/2.50/2.51 3.62/3.70/3.77 json 5.14 3.58 7.03 4.74/4.76/4.83 9.03/8.97/9.39 random 1.00 1.00 1.36 1.00 (stored raw) 1.36/1.36/1.34 pg_train_toast_dictionary() default dict_size = 65536. 9. 1 kB values (t_small.sh): a 1000-byte value alone in its row is stored uncompressed with pglz, lz4, zstd, lz4_dict and zstd_dict, with the default toast_tuple_target and with 128; 3000-byte values are compressed with each method. heapam.c:2266 calls the toaster only above TOAST_TUPLE_THRESHOLD.