notes.txt -- v19 of the zstd / TOAST dictionary series ===================================================== v19 applies on master as of 11591f273e6. v18 was prepared but never posted, so this answers the reviews of v17. The first part lists the changes for Hannu's and Manu's points by patch, with the test that covers each. The second part (R1..R16) lists what a further review pass and CI found after that. Then what I ran. Who raised what: H1..H5, Hm* Hannu, CAMT0RQQ8QyZwXX1+L6mWm=5R6ogic8PHQvF6_t=8U-sC8-xcNA@mail.gmail.com M1, M2 Manu, 179471842754.1333698.14079138634948894846@gmail.com R* found during my own review and CI runs 0002 Allow more than four TOAST compression methods ---------------------------------------------------- H1 t/007 assumed a little-endian 1-byte varlena header (0x01) for TOAST pointers. It now expects 0x80 on big-endian, using the same pack('L', 1) eq pack('V', 1) test that set_method_bits() uses. Checked by reading varatt.h (SET_VARTAG_1B_E); not run on BE. Hm1 A long pointer to an uncompressed value (is_compressed false, long tag) was accepted by the readers, which then returned the stored bytes as the value. New static toast_check_external_form() in detoast.c, called by toast_fetch_datum() and toast_fetch_datum_slice(), raises ERRCODE_DATA_CORRUPTED "toast value N is not compressed but has a long TOAST pointer"; toast_get_compression_id() calls the same function (R3 below). Test: t/007 case 5 (from 0003 on, when long pointers exist). M2 amcheck compares the method named by the pointer with the method in the compressed header at the start of the first chunk, since readers decode by the header and would never notice a pointer that disagrees with it. ToastedAttribute gains cmid. Runs with check_toast => true only, like the other chunk checks. Message: "toast value N has compression method id P in its TOAST pointer, but D in its data". Test: t/007 case 6 (method byte 4 on a zstd value), from 0004 on, where 4 (zstd_dict) is a real method and only this comparison catches it. (See R4 below for v19's change to the message.) 0003 Add zstd as a TOAST compression method -------------------------------------------- H1 The long-pointer part of t/007 has the same endianness fix. M1 t/007 gains a byte-3 case (the invalid ID). Below 0004 it is also above TOAST_LAST_COMPRESSION_ID; from 0004 on only the explicit method_byte == TOAST_INVALID_COMPRESSION_ID test in toast_compression_id_resolve() rejects it, and without it the Assert in toast_compression_id_needs_cmid_byte() fires (Manu's checks.txt section 2). ZSTD_TOAST_WINDOWLOG_MAX moved from toast_compression.c to toast_compression.h, so that 0004's dictionary context uses it too. t/007 now has 34 tests on 0003 and later (11 with plain pointers in 0002): plain bits 3, plain bits 2, long bits 0, long byte 0 (pglz), long byte 200, long byte 3, long pointer to uncompressed, long byte 4 (other method), and a well-formed row each for plain and long, which must not be reported and must read back. 0004 Add TOAST compression dictionaries for zstd and lz4 --------------------------------------------------------- H2 RelationInvalidateRelation() now frees rd_toastdictoids and rd_toastdictvalid, next to rd_amcache. When a subtransaction that installed a dictionary rolls back while an outer statement still has the relation open, the relcache entry is only marked invalid, and the outer statement kept compressing with the rolled-back dictionary: "ERROR: compression dictionary N does not exist". Test: compression_zstd_dict, the cmsub table (Hannu's trigger case, after a reconnect so the cache starts empty). Verified failing with the fix removed. H3a pg_train_toast_dictionary() accepts what ANALYZE accepts, through a new dict_check_privilege() mirroring vacuum_is_permitted_for_relation(): MAINTAIN on the table, or owning the database (for a table that is not shared). H3b The relation kind is checked from the pg_class tuple before the lock (so views, sequences, partitioned and foreign tables are not locked), and the privilege is checked again after table_open(), in case a REVOKE or an owner change committed while we waited. Test: compression_zstd_dict, a view is refused. H4 pg_train_toast_dictionary() takes at most a tenth of its sample budget (dict_size * DICT_SAMPLE_FACTOR) from any one value, and never more than DICT_SAMPLE_LIMIT (128kB). Before, a column of 128kB values could not be trained below a ~9kB dictionary (0 to 6 samples, ZDICT needs at least 7), and one wide value at the start of a table ended the scan with no sample at all. Test: compression_zstd_dict, the cmwide table (a 256kB value first, then 300 normal ones, dict_size 1024). H5 DECLARE_FOREIGN_KEY((tdrelid, tdattnum), pg_attribute, (attrelid, attnum)), as in pg_statistic.h; oidjoins.out updated. Dropped columns keep their pg_attribute row until DROP TABLE, which deletes the dictionaries first, so the key holds. Hm2 The dictionary decompression context gets the same ZSTD_d_windowLogMax as the plain one. Hm3 compression_zstd_dict now checks a textrange next to the composite and the array, as its comment said. 0005 Add automatic dictionary evolution during ANALYZE ------------------------------------------------------- H3c dictionary_evolution_max_retained now also makes pg_train_toast_dictionary() refuse to train another dictionary for a column that already has that many, unless the caller is a superuser: ERRCODE_CONFIGURATION_LIMIT_EXCEEDED, with errdetail naming the setting and errhint pointing at a rewrite. The setting stays PGC_SUSET. Docs: config.sgml and func-admin.sgml. Test: compression_dict_evolve, the evocap table (a MAINTAIN role trains twice with the cap at 2, is refused the third time; a superuser is not). Alternative I did not take: keep the function owner-only. A role with MAINTAIN can already VACUUM FULL the table, which is how the cap is cleared, so letting it train up to the cap seemed the more consistent rule. The cap lives in 0005; 0004 alone has none. Hm4 A non-concurrent REFRESH MATERIALIZED VIEW (with or without data) now resets tdtrainrows, as TRUNCATE does, since the new contents replace the old. matview.c calls toast_dictionary_retire(oid, true) after refresh_by_heap_swap(). Test: compression_dict_evolve, the evomv matview (ANALYZE records 300 rows, a plain REFRESH resets them to 0). Hm5 zstd_dict_compressed_total() and lz4_dict_compressed_total() check their compress bound (ZSTD_isError / AllocSizeIsValid) before palloc, as the store path does. Hm6 Without USE_ZSTD, toast_dictionary_maybe_evolve() returns before sampling, since both codecs train with zstd's trainer. Further fixes (R*) ------------------ 0002 R3 toast_check_external_form() is exported from detoast.c, and toast_get_compression_id() calls it instead of repeating the test, so pg_column_compression() now gives the same "is not compressed but has a long TOAST pointer" message as the readers. R4 The amcheck chunk-0 comparison reported a corrupt header in the data as the decoded ID, which is always 3 (invalid). It now reports an invalid header by what is stored, as the pointer checks do: "toast value N has invalid compression method id B in its data" (the method byte if the header claims the long form, else the bits). A valid header naming another method keeps "has compression method id P in its TOAST pointer, but D in its data". Checked by forging both on a cluster: byte 200 gives the first message, byte 4 on a zstd value the second. R5 The commit message said t/007 checks that reading the forged plain pointers "fails cleanly". At 0002 it checks that pg_column_compression() rejects them; reading them succeeds, because readers decode by the intact pglz header. Reworded. 0003 R1 t/007 test 33 read the uncompressed-long-pointer row with length(f). length() of an external text value is answered from va_rawsize without detoasting, so the test passed or failed by platform; it failed on CI's 32-bit, autoconf and MinGW jobs. It now selects f, which must detoast. Verified red with length(f) under LANG=C, green with f. pg_column_compression() on the same row is checked too (R3). R2 t/007's zstd rows used md5(), which FIPS-mode OpenSSL refuses (see 284cbaea7c4); they use sha256() now. Their raw size was ~6kB, below TOAST_TUPLE_THRESHOLD at 32kB blocks, so they would have stayed inline; the filler is 12000 bytes now. R6 t/007 case 6 accepted either of two messages, to work at both 0003 and 0004. It now expects exactly one in each: at 0003, ID 4 is an unknown method ("invalid compression method id 4"); from 0004 on it is zstd_dict, and the comparison message. R7 The comment and commit message said zstd shrinks the streaming buffer again by itself. That arrived in libzstd 1.4.5 (#2069) and the minimum is 1.4.0; both say "from 1.4.5" now. 0004 R8 lz4_cstream_reset() used LZ4_initStream() and LZ4_resetStream_fast(), both lz4 1.9.0+, but there is no minimum lz4 version (RHEL 8 has 1.8.3). LZ4_loadDict() resets the stream itself in every release I checked (1.7.5, 1.8.0, 1.8.3, 1.9.0, 1.10.0), so the stream is created once with LZ4_createStream() and reused; lz4_get_cstream() serves both callers. R9 The relkind check in dict_check_column() could never fire once pg_train_toast_dictionary() checked before the lock; it is gone, and the remaining one uses the house style: "cannot train a compression dictionary for relation "%s"" with errdetail_relkind_not_supported(). R10 The zstd_dict and lz4_dict cases of toast_compress_datum_rel() were twenty identical lines each; now one case. R11 func-admin.sgml: the database-owner half of the privilege rule moved here from 0005, where it had landed by mistake; the size range (256 to 1048576) and lz4's use of only the last 64kB of a dictionary are documented. R12 Comments: the relcache comments say invalidation frees the cache (they said rebuild); the windowLogMax comment says it only matters for streaming; the file header gives the reason for the design rather than referring to the list; "%d" for a method char is "%c"; long lines wrapped. pg_toast_dictionaries moved after pg_timezone_names in system-views.sgml. The view test filters on the schema as well as the table name, since compression_lz4_dict in the same parallel group also has a cmdict. 0005 R13 dictionary_evolution_max_retained counts the current dictionary, and a rewrite always keeps that one, so at its minimum of 1 a column could never be trained again. The minimum is 2. R14 At the cap, evolution restarted both gates, moving the growth baseline to the current row count, so after the VACUUM FULL the docs recommend it waited for the table to double again. It now restarts only the cooldown. Test: compression_dict_evolve, the evocapr table (cap reached after the table doubled; the baseline stays at 2000; after VACUUM FULL the next ANALYZE considers a candidate). Verified red with the old code (baseline 4000). R15 DictCompressRoutine.train is gone: both codecs pointed it at dict_train_zdict(), which both callers now call. dict_count() opens the catalog itself and sits next to dict_lookup(), so both cap checks are one line and the forward declaration is gone. analyze.c no longer repeats the attnum range check and the method test that toast_dictionary_maybe_evolve() does first. R16 Docs and messages: the max_retained descriptions mention manual training; the three PGC_SUSET settings carry the standard "Only superusers..." sentence; tdtrainrows/trained_rows say "or last considered for retraining", which is when evolution rewrites it; trained_at may be null; the manual-cap errcode is CONFIGURATION_LIMIT_EXCEEDED; the conf sample says 1d; a stray blank line in config.sgml is gone; the cap test asserts it reached 3 rather than "at most 3"; commit message corrected (the prefix sentence, the gates, what the test covers) and reflowed. Not changed, with reasons: - amcheck does not check inline compressed headers. That is a gap upstream too (amcheck skips non-external attributes); the readers reject a bad inline header since 0002. - The TOASTDICTOID catcache keeps whole dictionary rows, as STATRELATTINH does for pg_statistic. - Training reads whole values for non-pglz methods even though it keeps only a prefix; it can be cancelled, and the XXX comment notes the follow-up of training from ANALYZE's sample. What I ran ---------- Builds are cassert, -Og, zstd and lz4, on gcc 14.3, x86_64, unless noted. - Each patch on its own: -Werror build with no warnings, make check passes; pgindent and git diff --check are clean on each. - check-world on 0001-0005: 366 passed, 55 skipped, 0 failed. - Builds with zstd only, lz4 only, and neither: -Werror, no warnings, make check and t/007 pass (t/007 skips its zstd half without zstd). - clang 21: every changed .c file with -Wall -Werror, clean. - headerscheck and cpluspluscheck: clean (apart from the PLs, not configured here). - Docs: make check and make html. - CI (GitHub Actions, the cfbot task set) on the series before the R* fixes found R1 on 32-bit, autoconf and MinGW and was otherwise green (Linux 64-bit, Windows MSVC, CompilerWarnings); macOS did not run. I'll watch cfbot on this version. - t/007: 11 tests on 0002, 34 on 0003 and later; R1 and R14 were checked to fail without their fix, R4 by forging both header cases. Not done -------- - No big-endian run of t/007 (no hardware); H1 is by inspection. - No build against lz4 older than 1.9 (none here); R8 is by reading lz4.c of each release. - No new benchmark: nothing since v17 touches the paths measured then.