From 066c290083331d8b1eed895777b244e0dac9a799 Mon Sep 17 00:00:00 2001 From: Greg Burd Date: Sun, 1 Nov 2026 15:49:02 -0500 Subject: [PATCH v17 5/5] Add automatic dictionary evolution during ANALYZE A trained dictionary ages: as a column's data drifts, the dictionary that was best for the old data compresses the new data worse. Retraining was manual (pg_train_toast_dictionary). This adds an opt-in policy that lets ANALYZE retrain a column's dictionary when the data has changed enough to justify it, and only when a new dictionary actually compresses better. The policy is codec-agnostic; it rides the existing dictionary lifecycle and the DictCompressRoutine seam, so it serves zstd_dict and lz4_dict alike. None of it runs on the INSERT/UPDATE path. Opt-in per column, like n_distinct, via attribute options: dictionary_evolution (bool, default off) dictionary_evolution_growth (real, default 2.0) dictionary_evolution_min_rows (int, default 10000) and four GUCs, listed with the vacuum settings: dictionary_evolution_autovacuum (bool, default on, sighup) dictionary_evolution_min_gain (real, default 0.05, superuser) dictionary_evolution_min_interval (int seconds, default 86400, superuser) dictionary_evolution_max_retained (int, default 8, superuser) When ANALYZE processes an opted-in dictionary column, after it has the block sample it calls toast_dictionary_maybe_evolve(). That includes the ANALYZE that autovacuum runs, unless dictionary_evolution_autovacuum is off, so that a column which opts in keeps its dictionary current without a scheduled job. Nothing is done in a read-only transaction: ANALYZE is allowed there because its statistics do not change what a dump would contain, but a new dictionary changes how every later value is stored. - Gates. Do nothing unless the live row count is at least min_rows, has grown by the growth factor since the current dictionary was trained (pg_toast_dictionary.tdtrainrows), and the current dictionary is older than min_interval (tdtraintime). If the column already has max_retained dictionaries, decline with a LOG and suggest a rewrite, since retirement only happens on rewrite, and restart the gates so that the LOG is repeated at most once per interval. TRUNCATE resets tdtrainrows, so that growth is measured from the table's new contents. Data that changes in place without the table growing does not trigger retraining; pg_train_toast_dictionary() is there for that. A dictionary trained by hand before the table's row count was known records 0 rows; the first ANALYZE then records its count as the baseline instead of treating any count as growth, even while the table is below min_rows. The gates read only the dictionary's row; its bytes, which may be toasted, are fetched only for the held-out test. - Candidate. Build samples from ANALYZE's own rows[] (for inline values no extra I/O; a value stored out of line is fetched here, which ANALYZE skipped), with the same prefix and skip rules as the manual trainer. Only pglz can decompress a prefix without decompressing the whole value, so values over 1MB are skipped rather than fetched and decompressed whole. Bound the sample the same way as the manual trainer too: at most DICT_MAX_SAMPLES values and a few MB, visiting rows[] on a stride so the bounded sample still spans the table. The loop honors the cost-based vacuum delay, as the rest of ANALYZE does. Fewer than 16 values are not trained on. Hold out every fifth value and train a candidate on the rest through the codec's train() routine. The candidate dictionary size is derived from the sample. - Adopt only on a win. Compress the held-out values with the current dictionary and with the candidate through the codec's compressed_total() routine, which digests each dictionary once, and adopt the candidate only if it is at least min_gain smaller on data it was not trained on. The first dictionary of a column is adopted unconditionally. When a candidate is not adopted, the gates restart anyway, so the following ANALYZEs do not train again on the same data. - Failure handling. The evaluation runs in an internal subtransaction: an error rolls back whatever the failed step held and is reported as a WARNING with the original error code, and ANALYZE goes on. A query cancel is re-thrown, so it stops ANALYZE as it would anywhere else. Two pg_toast_dictionary columns support the gates: tdtrainrows (the reltuples estimate at training time) and tdtraintime. dict_install() writes them, and pg_upgrade carries both over. pg_toast_dictionaries shows them as trained_rows and trained_at (a timestamptz). The regression test now checks that values compressed with a dictionary that evolution has replaced still read back, and that a rewrite then removes it, and covers TRUNCATE and read-only transactions. Reviewed-by: Manuel Reyes Bravo Reviewed-by: Hannu Krosing Discussion: https://postgr.es/m/CAFAfj_GGY4W81XjtYLvYOhqPtvfZbBNj=YRS6Mq_0SA9AqEwoQ@mail.gmail.com --- doc/src/sgml/catalogs.sgml | 36 +- doc/src/sgml/config.sgml | 85 +++ doc/src/sgml/ref/alter_table.sgml | 37 +- doc/src/sgml/system-views.sgml | 21 + src/backend/access/common/reloptions.c | 35 +- src/backend/access/common/toast_dictionary.c | 657 +++++++++++++++++- src/backend/catalog/heap.c | 2 +- src/backend/catalog/system_views.sql | 6 +- src/backend/commands/analyze.c | 21 + src/backend/commands/repack.c | 2 +- src/backend/commands/tablecmds.c | 2 +- src/backend/utils/misc/guc_parameters.dat | 34 + src/backend/utils/misc/guc_tables.c | 1 + src/backend/utils/misc/postgresql.conf.sample | 9 + src/bin/pg_dump/pg_dump.c | 8 +- src/bin/psql/tab-complete.in.c | 4 +- src/include/access/toast_dictionary.h | 45 +- src/include/catalog/pg_proc.dat | 2 +- src/include/catalog/pg_toast_dictionary.h | 6 +- src/include/utils/attoptcache.h | 3 + .../expected/compression_dict_evolve.out | 292 ++++++++ .../expected/compression_dict_evolve_1.out | 11 + src/test/regress/expected/rules.out | 7 +- src/test/regress/parallel_schedule | 2 +- .../regress/sql/compression_dict_evolve.sql | 188 +++++ 25 files changed, 1476 insertions(+), 40 deletions(-) create mode 100644 src/test/regress/expected/compression_dict_evolve.out create mode 100644 src/test/regress/expected/compression_dict_evolve_1.out create mode 100644 src/test/regress/sql/compression_dict_evolve.sql diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 85e19ed3664..21adaefa3f3 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -8527,11 +8527,12 @@ SCRAM-SHA-256$<iteration count>:&l compression dictionaries used by the zstd_dict and lz4_dict TOAST compression methods. Each dictionary belongs to one column of one table; it is created by - pg_train_toast_dictionary, and removed when its table - is dropped, or when a rewrite or truncation of the table has removed every - value that used it. The dictionaries of a dropped column stay until the - next rewrite of the table. Values compressed with a dictionary are never - stored outside the column that owns it. + pg_train_toast_dictionary or by automatic dictionary + evolution, and removed when its table is dropped, or when a rewrite or + truncation of the table has removed every value that used it. The + dictionaries of a dropped column stay until the next rewrite of the + table. Values compressed with a dictionary are never stored outside the + column that owns it. @@ -8596,6 +8597,31 @@ SCRAM-SHA-256$<iteration count>:&l + + + tdtrainrows int8 + + + Estimated number of rows in the table when the dictionary was trained, + or 0 if not known; used by automatic dictionary evolution + + + + + + tdtraintime int8 + + + Time when the dictionary was trained, or last considered for + retraining by automatic dictionary evolution, as a + timestamp with time zone in its internal representation + (microseconds since 2000-01-01 00:00:00 UTC); the + trained_at column of + pg_toast_dictionaries + shows it as a timestamp with time zone + + + tddict bytea diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 188c513ccab..e6f67dfb878 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -10094,6 +10094,90 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv; + + + dictionary_evolution_autovacuum (boolean) + + dictionary_evolution_autovacuum configuration parameter + + + + + Allows the ANALYZE that autovacuum runs to retrain + the TOAST compression + dictionary of a column that opts into automatic dictionary + evolution (with the dictionary_evolution + attribute option). When off, only a manual + ANALYZE does. Retraining samples the column, + trains a candidate dictionary and compares it with the current one, + which takes some CPU time and memory per column, and installs a new + dictionary only when it compresses better. The default is + on. This parameter can only be set in the + postgresql.conf file or on the server command + line. + + + + + + dictionary_evolution_max_retained (integer) + + dictionary_evolution_max_retained configuration parameter + + + + + The maximum number of TOAST + compression dictionaries kept per column by automatic dictionary + evolution (enabled with the per-column + dictionary_evolution attribute option). When a + column reaches this many dictionaries, evolution stops retraining it + and logs a message; a table rewrite + (VACUUM FULL, CLUSTER, or + REPACK) retires the superseded dictionaries so + evolution can resume. The default is 8. + + + + + + dictionary_evolution_min_gain (floating point) + + dictionary_evolution_min_gain configuration parameter + + + + + The minimum fractional reduction in compressed size a newly trained + candidate TOAST compression + dictionary must achieve, measured on a held-out sample, before + automatic evolution adopts it in place of the current dictionary. + For example, the default of 0.05 requires the + candidate to compress the held-out sample to at most 95% of the size + the current dictionary achieves. A larger value makes evolution more + conservative. + + + + + + dictionary_evolution_min_interval (integer) + + dictionary_evolution_min_interval configuration parameter + + + + + The minimum time that must pass after a column's + TOAST compression dictionary was + trained, or after a retrained candidate last failed to win, before + automatic evolution will retrain it again, acting as a cooldown that + bounds churn. If this value is specified without units, + it is taken as seconds. The default is one day + (86400 seconds). + + + @@ -10563,6 +10647,7 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv; + temp_tablespaces (string) diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml index 601d1c2089f..bffa4a15059 100644 --- a/doc/src/sgml/ref/alter_table.sgml +++ b/doc/src/sgml/ref/alter_table.sgml @@ -382,9 +382,9 @@ WITH ( MODULUS numeric_literal, REM RESET ( attribute_option [, ... ] ) - This form sets or resets per-attribute options. Currently, the only - defined per-attribute options are n_distinct and - n_distinct_inherited, which override the + This form sets or resets per-attribute options. Two of the defined + per-attribute options, n_distinct and + n_distinct_inherited, override the number-of-distinct-values estimates made by subsequent ANALYZE operations. n_distinct affects the statistics for the @@ -404,6 +404,37 @@ WITH ( MODULUS numeric_literal, REM PostgreSQL query planner, refer to . + + Three further per-attribute options control automatic + TOAST compression dictionary + evolution, and apply only to columns that use a dictionary compression + method (zstd_dict or lz4_dict). + dictionary_evolution (boolean, default + false) enables, for the column, retraining of its + compression dictionary from the sample gathered by a subsequent + ANALYZE; a newly + trained dictionary is adopted only if it wins a held-out size test by + the margin set by . + dictionary_evolution_growth (floating point, default + 2.0, minimum 1.0) is the factor by + which the column's live-row count must grow since the current dictionary + was trained, or since a retrained candidate last failed to win, before + retraining is considered. + dictionary_evolution_min_rows (integer, default + 10000) is a floor below which the column is never + retrained. Like other per-attribute options, these are not inherited, + so on a partitioned table they must be set on the leaf partitions. + + + Retraining is triggered by the growth of the table, so data that + changes in place without the table growing does not trigger it; run + pg_train_toast_dictionary for that case. + TRUNCATE resets the baseline, so the table's + growth is measured again from its new contents. Retraining also + happens in the ANALYZE that autovacuum runs, unless + is off, and + never in a read-only transaction. + Changing per-attribute options acquires a SHARE UPDATE EXCLUSIVE lock. diff --git a/doc/src/sgml/system-views.sgml b/doc/src/sgml/system-views.sgml index e2b161777a9..b1067e080ec 100644 --- a/doc/src/sgml/system-views.sgml +++ b/doc/src/sgml/system-views.sgml @@ -5367,6 +5367,27 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx Size of the dictionary in bytes + + + + trained_rows int8 + + + Estimated number of rows in the table when the dictionary was trained, + or 0 if not known; automatic dictionary evolution measures the + table's growth from it + + + + + + trained_at timestamp with time zone + + + Time when the dictionary was trained, or last considered for + retraining by automatic dictionary evolution + + diff --git a/src/backend/access/common/reloptions.c b/src/backend/access/common/reloptions.c index ea9a0417909..44b41e91d6d 100644 --- a/src/backend/access/common/reloptions.c +++ b/src/backend/access/common/reloptions.c @@ -153,6 +153,15 @@ static relopt_bool boolRelOpts[] = }, true }, + { + { + "dictionary_evolution", + "Enables automatic retraining of this column's TOAST compression dictionary during ANALYZE.", + RELOPT_KIND_ATTRIBUTE, + ShareUpdateExclusiveLock + }, + false + }, /* list terminator */ {{NULL}} }; @@ -412,6 +421,15 @@ static relopt_int intRelOpts[] = }, -1, 0, 1024 }, + { + { + "dictionary_evolution_min_rows", + "Minimum live-row count before a column's TOAST compression dictionary is eligible for automatic retraining.", + RELOPT_KIND_ATTRIBUTE, + ShareUpdateExclusiveLock + }, + 10000, 0, INT_MAX + }, /* list terminator */ {{NULL}} @@ -501,6 +519,15 @@ static relopt_real realRelOpts[] = }, 0, -1.0, DBL_MAX }, + { + { + "dictionary_evolution_growth", + "Factor by which a column's live-row count must grow since its TOAST compression dictionary was trained before automatic retraining is eligible.", + RELOPT_KIND_ATTRIBUTE, + ShareUpdateExclusiveLock + }, + 2.0, 1.0, DBL_MAX + }, { { "vacuum_cleanup_index_scale_factor", @@ -2418,7 +2445,13 @@ attribute_reloptions(Datum reloptions, bool validate) { static const relopt_parse_elt tab[] = { {"n_distinct", RELOPT_TYPE_REAL, offsetof(AttributeOpts, n_distinct)}, - {"n_distinct_inherited", RELOPT_TYPE_REAL, offsetof(AttributeOpts, n_distinct_inherited)} + {"n_distinct_inherited", RELOPT_TYPE_REAL, offsetof(AttributeOpts, n_distinct_inherited)}, + {"dictionary_evolution", RELOPT_TYPE_BOOL, + offsetof(AttributeOpts, dictionary_evolution)}, + {"dictionary_evolution_growth", RELOPT_TYPE_REAL, + offsetof(AttributeOpts, dictionary_evolution_growth)}, + {"dictionary_evolution_min_rows", RELOPT_TYPE_INT, + offsetof(AttributeOpts, dictionary_evolution_min_rows)} }; return (bytea *) build_reloptions(reloptions, validate, diff --git a/src/backend/access/common/toast_dictionary.c b/src/backend/access/common/toast_dictionary.c index 2f3781c097a..30971a64475 100644 --- a/src/backend/access/common/toast_dictionary.c +++ b/src/backend/access/common/toast_dictionary.c @@ -67,26 +67,31 @@ #include "access/tableam.h" #include "access/toast_compression.h" #include "access/toast_dictionary.h" +#include "access/xact.h" #include "catalog/catalog.h" #include "catalog/indexing.h" #include "catalog/objectaddress.h" #include "catalog/pg_class.h" #include "catalog/pg_toast_dictionary.h" #include "catalog/pg_type.h" +#include "commands/vacuum.h" #include "executor/tuptable.h" #include "fmgr.h" #include "lib/stringinfo.h" #include "miscadmin.h" #include "nodes/bitmapset.h" #include "utils/acl.h" +#include "utils/attoptcache.h" #include "utils/builtins.h" #include "utils/fmgroids.h" #include "utils/inval.h" #include "utils/lsyscache.h" #include "utils/memutils.h" #include "utils/rel.h" +#include "utils/resowner.h" #include "utils/snapmgr.h" #include "utils/syscache.h" +#include "utils/timestamp.h" #define NO_ZSTD_SUPPORT() \ ereport(ERROR, \ @@ -102,10 +107,54 @@ #define DICT_MIN_SIZE 256 #define DICT_MAX_SIZE (1024 * 1024) +#define DICT_DEFAULT_SIZE (64 * 1024) /* pg_train_toast_dictionary default */ #define DICT_MAX_SAMPLES 10000 #define DICT_SAMPLE_LIMIT (128 * 1024) /* zstd recommends samples totalling about 100 times the dictionary size */ #define DICT_SAMPLE_FACTOR 100 +/* training bytes automatic evolution samples, enough for the largest candidate */ +#define DICT_EVOLVE_SAMPLE_BYTES ((size_t) DICT_DEFAULT_SIZE * DICT_SAMPLE_FACTOR) +/* fewer sampled values than this are not enough to train on */ +#define DICT_EVOLVE_MIN_SAMPLES 16 +/* larger values are not sampled by automatic evolution (see there) */ +#define DICT_EVOLVE_VALUE_LIMIT (8 * DICT_SAMPLE_LIMIT) + +/* + * GUCs for automatic dictionary evolution. Codec-agnostic policy knobs; see + * toast_dictionary_maybe_evolve(). Declared in toast_dictionary.h. + */ +double dictionary_evolution_min_gain = 0.05; +int dictionary_evolution_min_interval = 86400; +int dictionary_evolution_max_retained = 8; +bool dictionary_evolution_autovacuum = true; + +/* + * Train a dictionary with zstd's ZDICT trainer. Shared by both codecs: + * lz4_dict dictionaries are also trained by ZDICT (lz4 ships no trainer of + * its own), so lz4_dict_routine.train points here too. Returns the trained + * dictionary's length, written into dictbuf (which must hold max_dict_bytes), + * or 0 on failure -- a decline to retrain, not an error. When the server is + * built without zstd, training is unavailable for both codecs and this + * returns 0. + */ +static size_t +dict_train_zdict(const char *samples, const size_t *sizes, int nsamples, + size_t max_dict_bytes, void *dictbuf) +{ +#ifndef USE_ZSTD + return 0; +#else + size_t dictlen; + + if (nsamples == 0) + return 0; + dictlen = ZDICT_trainFromBuffer(dictbuf, max_dict_bytes, samples, sizes, + nsamples); + if (ZDICT_isError(dictlen)) + return 0; + return dictlen; +#endif +} /* * The relation owning the dictionaries of rel's columns. During a table @@ -567,6 +616,65 @@ zstd_dict_decompress_datum_slice(const varlena *value, int32 slicelength) return result; } +/* + * Total compressed size of the held-out values against raw dictionary bytes, + * for the held-out test of dictionary evolution. Digests the bytes into a + * one-shot CDict once for all the values (not the per-backend digested cache, + * which keys on OID and a candidate has no OID yet) and returns the total + * compressed byte count without storing anything. Returns 0 on any zstd + * failure so the caller can treat it as "no estimate." + */ +static size_t +zstd_dict_compressed_total(const char *dict, size_t dictlen, + const char *vals, const size_t *sizes, int nvals) +{ +#ifndef USE_ZSTD + return 0; +#else + ZSTD_CDict *cd; + size_t total = 0; + size_t maxlen = 0; + char *out; + + for (int i = 0; i < nvals; i++) + maxlen = Max(maxlen, sizes[i]); + if (dict_cctx == NULL && (dict_cctx = ZSTD_createCCtx()) == NULL) + elog(ERROR, "could not create zstd compression context"); + + /* + * Allocate the output buffer before creating the CDict, so no palloc + * (which can ereport on OOM) sits between ZSTD_createCDict() and + * ZSTD_freeCDict(): toast_dictionary_maybe_evolve() catches errors in a + * subtransaction, and a longjmp from between the two would leak the + * malloc'd CDict for the life of the backend. + */ + out = palloc(ZSTD_compressBound(maxlen)); + cd = ZSTD_createCDict(dict, dictlen, ZSTD_CLEVEL_DEFAULT); + if (cd == NULL) + { + pfree(out); + return 0; + } + for (int i = 0; i < nvals; i++) + { + size_t clen = ZSTD_compress_usingCDict(dict_cctx, out, + ZSTD_compressBound(maxlen), + vals, sizes[i], cd); + + if (ZSTD_isError(clen)) + { + total = 0; + break; + } + total += clen; + vals += sizes[i]; + } + ZSTD_freeCDict(cd); + pfree(out); + return total; +#endif +} + #ifdef USE_LZ4 /* * Per-backend cache of the raw dictionary bytes most recently used, keyed by @@ -756,6 +864,45 @@ lz4_dict_decompress_datum_slice(const varlena *value, int32 slicelength) return result; } +/* + * Total compressed size of the held-out values against raw dictionary bytes, + * for the held-out test of dictionary evolution. Loads the bytes into a fresh + * lz4 stream for each value, as compression does, and returns the total + * compressed byte count without storing anything. Returns 0 on any lz4 + * failure. + */ +static size_t +lz4_dict_compressed_total(const char *dict, size_t dictlen, + const char *vals, const size_t *sizes, int nvals) +{ +#ifndef USE_LZ4 + return 0; +#else + size_t total = 0; + + for (int i = 0; i < nvals; i++) + { + int32 bound = LZ4_compressBound(sizes[i]); + char *out; + int32 clen; + + if (bound <= 0) + return 0; + /* static stream reset per value; no malloc to leak on OOM */ + out = palloc(bound); + lz4_cstream_reset(); + LZ4_loadDict(lz4_cstream, dict, dictlen); + clen = LZ4_compress_fast_continue(lz4_cstream, vals, out, sizes[i], bound, 1); + pfree(out); + if (clen <= 0) + return 0; + total += clen; + vals += sizes[i]; + } + return total; +#endif +} + /* * The lz4 dictionary codec. See DictCompressRoutine in toast_dictionary.h. */ @@ -763,6 +910,8 @@ static const DictCompressRoutine lz4_dict_routine = { .compress = lz4_dict_compress_datum, .decompress = lz4_dict_decompress_datum, .decompress_slice = lz4_dict_decompress_datum_slice, + .train = dict_train_zdict, + .compressed_total = lz4_dict_compressed_total, }; /* @@ -773,6 +922,8 @@ static const DictCompressRoutine zstd_dict_routine = { .compress = zstd_dict_compress_datum, .decompress = zstd_dict_decompress_datum, .decompress_slice = zstd_dict_decompress_datum_slice, + .train = dict_train_zdict, + .compressed_total = zstd_dict_compressed_total, }; /* @@ -871,11 +1022,13 @@ RemoveToastDictionaries(Oid relid, AttrNumber attnum) * that toast_dictionary_keep() rejects: non-current ones, and all of those * of a column that no longer uses a dictionary method. The rewrite * recompressed those values, so delete such dictionaries. TRUNCATE calls - * this too, having removed every value. Caller holds AccessExclusiveLock on - * relid. + * this too, having removed every value, with truncated set: the row count + * recorded with a kept dictionary no longer describes the table, so it is + * reset to "not known", and automatic evolution takes the table's next count + * as its baseline. Caller holds AccessExclusiveLock on relid. */ void -toast_dictionary_retire(Oid relid) +toast_dictionary_retire(Oid relid, bool truncated) { Relation drel; SysScanDesc scan; @@ -889,6 +1042,14 @@ toast_dictionary_retire(Oid relid) if (!d->tdcurrent || !dict_column_uses_dict(relid, d->tdattnum)) CatalogTupleDelete(drel, &tup->t_self); + else if (truncated && d->tdtrainrows != 0) + { + HeapTuple newtup = heap_copytuple(tup); + + ((Form_pg_toast_dictionary) GETSTRUCT(newtup))->tdtrainrows = 0; + CatalogTupleUpdate(drel, &newtup->t_self, newtup); + heap_freetuple(newtup); + } } systable_endscan(scan); table_close(drel, RowExclusiveLock); @@ -897,9 +1058,13 @@ toast_dictionary_retire(Oid relid) /* * Insert a dictionary for rel.attnum and make it current, retiring the * previous current one. dictid may be given (binary upgrade) or chosen here. + * trainrows and traintime are the row count and time to record for the + * evolution gates; a trainrows of 0 means the row count is not known yet (see + * toast_dictionary_maybe_evolve()). */ static Oid dict_install(Relation rel, AttrNumber attnum, Oid dictid, bool current, + int64 trainrows, TimestampTz traintime, const void *dict, size_t dictlen) { Relation drel; @@ -937,6 +1102,9 @@ dict_install(Relation rel, AttrNumber attnum, Oid dictid, bool current, ObjectIdGetDatum(RelationGetRelid(rel)); values[Anum_pg_toast_dictionary_tdattnum - 1] = Int16GetDatum(attnum); values[Anum_pg_toast_dictionary_tdcurrent - 1] = BoolGetDatum(current); + values[Anum_pg_toast_dictionary_tdtrainrows - 1] = Int64GetDatum(trainrows); + values[Anum_pg_toast_dictionary_tdtraintime - 1] = + TimestampTzGetDatum(traintime); values[Anum_pg_toast_dictionary_tddict - 1] = PointerGetDatum(dictval); tup = heap_form_tuple(RelationGetDescr(drel), values, nulls); CatalogTupleInsert(drel, tup); @@ -957,6 +1125,36 @@ dict_install(Relation rel, AttrNumber attnum, Oid dictid, bool current, return dictid; } +/* + * Record new values for the evolution gates on the current dictionary of + * rel.attnum, without changing the dictionary itself. + */ +static void +dict_set_gates(Relation rel, AttrNumber attnum, int64 trainrows, + TimestampTz traintime) +{ + Relation drel; + SysScanDesc scan; + HeapTuple tup; + + drel = table_open(ToastDictionaryRelationId, RowExclusiveLock); + tup = dict_lookup(RelationGetRelid(rel), attnum, true, drel, &scan); + if (tup) + { + HeapTuple newtup = heap_copytuple(tup); + Form_pg_toast_dictionary d = (Form_pg_toast_dictionary) GETSTRUCT(newtup); + + d->tdtrainrows = trainrows; + d->tdtraintime = traintime; + CatalogTupleUpdate(drel, &newtup->t_self, newtup); + heap_freetuple(newtup); + } + systable_endscan(scan); + table_close(drel, RowExclusiveLock); + + CommandCounterIncrement(); +} + #ifdef USE_ZSTD /* Checks for pg_train_toast_dictionary(); returns attnum */ static AttrNumber @@ -1013,9 +1211,9 @@ dict_check_column(Relation rel, const char *colname) * Takes ShareUpdateExclusiveLock, like ANALYZE: DML continues, concurrent * training and DDL wait. * - * XXX: this samples the first DICT_MAX_SAMPLES rows of a seqscan. Training - * from ANALYZE's block sample is the natural follow-up once the lifecycle is - * agreed on. + * Automatic evolution (toast_dictionary_maybe_evolve) trains from ANALYZE's + * block sample instead; this function keeps its own bounded seqscan for the + * explicit, user-requested case. */ Datum pg_train_toast_dictionary(PG_FUNCTION_ARGS) @@ -1041,6 +1239,7 @@ pg_train_toast_dictionary(PG_FUNCTION_ARGS) void *dictbuf; size_t dictlen; Oid dictid; + int64 trainrows; HeapTuple classtup; Form_pg_class classform; AclResult aclresult; @@ -1125,10 +1324,10 @@ pg_train_toast_dictionary(PG_FUNCTION_ARGS) ExecDropSingleTupleTableSlot(slot); dictbuf = palloc(dict_size); - dictlen = nsamples == 0 ? 0 : - ZDICT_trainFromBuffer(dictbuf, dict_size, samples.data, sizes, - nsamples); - if (nsamples == 0 || ZDICT_isError(dictlen)) + dictlen = dict_routine_for_method(TupleDescAttr(RelationGetDescr(rel), + attnum - 1)->attcompression) + ->train(samples.data, sizes, nsamples, dict_size, dictbuf); + if (dictlen == 0) { ereport(NOTICE, (errmsg("could not train a compression dictionary for column \"%s\" from %d values", @@ -1139,7 +1338,14 @@ pg_train_toast_dictionary(PG_FUNCTION_ARGS) PG_RETURN_NULL(); } - dictid = dict_install(rel, attnum, InvalidOid, true, dictbuf, dictlen); + /* + * reltuples is -1 on a table that has never been vacuumed or analyzed. + * Record 0 then, which tells evolution that the row count at training + * time is not known. + */ + trainrows = rel->rd_rel->reltuples < 0 ? 0 : (int64) rel->rd_rel->reltuples; + dictid = dict_install(rel, attnum, InvalidOid, true, + trainrows, GetCurrentTimestamp(), dictbuf, dictlen); MemoryContextSwitchTo(oldcxt); MemoryContextDelete(workcxt); @@ -1149,12 +1355,431 @@ pg_train_toast_dictionary(PG_FUNCTION_ARGS) #endif } +/* + * Collect the current dictionary row of rel.attnum, if any, returning its OID + * and, when wantbytes, a palloc'd copy of its dictionary bytes (caller frees). + * Also returns tdtrainrows and tdtraintime for the evolution gates. Returns + * InvalidOid when the column has no current dictionary. + */ +static Oid +dict_current_info(Relation drel, Oid relid, AttrNumber attnum, + int64 *trainrows, int64 *traintime, + bytea **dictbytes, bool wantbytes) +{ + SysScanDesc scan; + HeapTuple tup; + Oid result = InvalidOid; + + *trainrows = 0; + *traintime = 0; + if (dictbytes) + *dictbytes = NULL; + tup = dict_lookup(relid, attnum, true, drel, &scan); + if (tup) + { + Form_pg_toast_dictionary d = (Form_pg_toast_dictionary) GETSTRUCT(tup); + + result = d->oid; + *trainrows = d->tdtrainrows; + *traintime = d->tdtraintime; + if (wantbytes) + *dictbytes = DatumGetByteaPCopy(SysCacheGetAttrNotNull(TOASTDICTOID, tup, + Anum_pg_toast_dictionary_tddict)); + } + systable_endscan(scan); + return result; +} + +/* Count dictionaries (current + non-current) for rel.attnum */ +static int +dict_count(Relation drel, Oid relid, AttrNumber attnum) +{ + SysScanDesc scan; + HeapTuple tup; + int n = 0; + + tup = dict_lookup(relid, attnum, false, drel, &scan); + for (; HeapTupleIsValid(tup); tup = systable_getnext(scan)) + n++; + systable_endscan(scan); + return n; +} + +/* + * A stride for visiting all n elements of an array once each, in an order + * spread over the whole array: index (k * stride) % n for k = 0 .. n-1. The + * stride is coprime with n, so every index is visited, and close to n times + * the golden ratio, so any prefix of the visits samples the array evenly. + */ +static int +dict_spread_stride(int n) +{ + int stride; + + if (n <= 2) + return 1; + stride = (int) (n * 0.6180339887); + for (;;) + { + int a = n, + b = stride; + + while (b != 0) + { + int t = a % b; + + a = b; + b = t; + } + if (a == 1) + return stride; + stride++; + } +} + +/* + * toast_dictionary_maybe_evolve + * + * Automatic, best-effort dictionary retraining during ANALYZE, including the + * ANALYZE that autovacuum runs, unless dictionary_evolution_autovacuum is + * off. It holds ANALYZE's ShareUpdateExclusiveLock, like the manual trainer, + * reuses ANALYZE's already-fetched sample rows, and installs a new current + * dictionary only when a held-out size test says a candidate beats the + * current one by the configured margin. The write path is never involved. + * Nothing is done in a read-only transaction: a new dictionary changes how + * every later value is stored, which is more than ANALYZE's statistics. The work runs in an internal subtransaction: an error rolls it + * back, releasing whatever the failed step held (a TOAST scan, buffer pins, + * a half-done dict_install()), is reported as a WARNING, and ANALYZE goes on. + * A query cancel or a pending interrupt is re-thrown instead, so ANALYZE stops + * as it would anywhere else. + */ +void +toast_dictionary_maybe_evolve(Relation onerel, AttrNumber attnum, + HeapTuple *rows, int numrows, double totalrows) +{ + AttributeOpts *aopt; + Form_pg_attribute att; + const DictCompressRoutine *routine; + MemoryContext evolvecxt, + oldcxt; + ResourceOwner oldowner; + TupleDesc tupdesc = RelationGetDescr(onerel); + + /* + * (a) a table that can own dictionaries (as for the manual trainer), + * opted in, and still a column that compresses with a dictionary? + */ + if ((onerel->rd_rel->relkind != RELKIND_RELATION && + onerel->rd_rel->relkind != RELKIND_MATVIEW) || + IsCatalogRelation(onerel)) + return; + if (attnum <= 0 || attnum > tupdesc->natts) + return; + att = TupleDescAttr(tupdesc, attnum - 1); + if (!dict_attr_uses_dict(att)) + return; + aopt = get_attribute_options(RelationGetRelid(onerel), attnum); + if (aopt == NULL || !aopt->dictionary_evolution) + return; + if (XactReadOnly) + return; + if (AmAutoVacuumWorkerProcess() && !dictionary_evolution_autovacuum) + return; + + routine = dict_routine_for_method(att->attcompression); + + /* + * Everything below runs in a short-lived context so that samples, the + * candidate dictionary, and fetched current-dictionary bytes are all + * freed on return, and in an internal subtransaction so that a codec or + * catalog error degrades to "no evolution this round" rather than + * aborting ANALYZE, without leaking what the failed step held. + */ + oldcxt = CurrentMemoryContext; + oldowner = CurrentResourceOwner; + evolvecxt = AllocSetContextCreate(oldcxt, + "Dictionary evolution", + ALLOCSET_DEFAULT_SIZES); + + BeginInternalSubTransaction(NULL); + MemoryContextSwitchTo(evolvecxt); + + PG_TRY(); + { + Relation drel; + Oid curid; + int64 currows; + int64 curtime; + bytea *curbytes = NULL; + int ndicts; + StringInfoData train; + StringInfoData held; + size_t *trainsizes; + size_t *heldsizes; + int ntrain = 0; + int nheld = 0; + int stride; + size_t samplebytes = 0; + int32 dictsize; + void *candbuf; + size_t candlen; + bool adopt; + + /* + * Read only what the gates need here; the dictionary bytes (which may + * be toasted) are fetched only if the held-out test needs them. + */ + drel = table_open(ToastDictionaryRelationId, AccessShareLock); + curid = dict_current_info(drel, RelationGetRelid(onerel), attnum, + &currows, &curtime, NULL, false); + table_close(drel, AccessShareLock); + + /* + * The dictionary was trained before the table's row count was known + * (pg_train_toast_dictionary() on a table never vacuumed or + * analyzed). Take this count as the baseline for growth rather than + * treating any count as growth. Do it before the row floor, so that + * the baseline is the size the table had then. + */ + if (OidIsValid(curid) && currows <= 0) + { + if (totalrows > 0) + dict_set_gates(onerel, attnum, (int64) totalrows, curtime); + goto done; + } + + /* (b) growth, floor, and cooldown gates */ + if (totalrows < (double) aopt->dictionary_evolution_min_rows) + goto done; + if (OidIsValid(curid)) + { + if (totalrows < (double) currows * aopt->dictionary_evolution_growth) + goto done; + if (curtime != 0 && + !TimestampDifferenceExceedsSeconds((TimestampTz) curtime, + GetCurrentTimestamp(), + dictionary_evolution_min_interval)) + goto done; + } + + /* (c) cap gate */ + drel = table_open(ToastDictionaryRelationId, AccessShareLock); + ndicts = dict_count(drel, RelationGetRelid(onerel), attnum); + table_close(drel, AccessShareLock); + if (ndicts >= dictionary_evolution_max_retained) + { + ereport(LOG, + (errmsg("automatic dictionary evolution for column \"%s\" of relation \"%s.%s\" stopped: %d dictionaries are retained", + NameStr(att->attname), + get_namespace_name(RelationGetNamespace(onerel)), + RelationGetRelationName(onerel), + ndicts), + errdetail("The limit is set by \"%s\".", + "dictionary_evolution_max_retained"), + errhint("A rewrite of the table, such as VACUUM FULL, removes the dictionaries no value uses any more."))); + /* restart the gates, so that this is reported once per interval */ + goto declined; + } + + /* + * (d) build samples from the ANALYZE rows[]. Split deterministically + * into an 80% training set and a 20% held-out set (every 5th sample + * is held out), using the same predicates as the manual trainer: skip + * nulls and values under 8 bytes, detoast a 128kB prefix. + * + * Bound the sample as the manual trainer does: at most + * DICT_MAX_SAMPLES values, and as many training bytes as the largest + * candidate wants (DICT_SAMPLE_FACTOR times DICT_DEFAULT_SIZE); the + * held-out values are every fifth one, so about a quarter of that + * again. Values stored out of line are fetched from the TOAST table + * here, so the bound also limits that I/O, and the loop honors the + * cost-based vacuum delay as the rest of ANALYZE does. rows[] is in + * physical order and either bound may stop the loop early, so visit + * the rows in an order spread over the whole array (see + * dict_spread_stride()) rather than from the start. + */ + initStringInfo(&train); + initStringInfo(&held); + trainsizes = palloc_array(size_t, Min(numrows, DICT_MAX_SAMPLES)); + heldsizes = palloc_array(size_t, Min(numrows, DICT_MAX_SAMPLES)); + stride = dict_spread_stride(numrows); + for (int k = 0, i = 0; k < numrows; k++, i = (int) (((int64) i + stride) % numrows)) + { + bool isnull; + Datum d; + varlena *v; + size_t len; + + vacuum_delay_point(true); + d = heap_getattr(rows[i], attnum, tupdesc, &isnull); + if (isnull) + continue; + + /* + * Only pglz can decompress a prefix of a value without + * decompressing it whole, so skip values much larger than the + * prefix we want rather than fetch and decompress them in full. + */ + if (toast_raw_datum_size(d) > (Size) DICT_EVOLVE_VALUE_LIMIT) + continue; + v = detoast_attr_slice((varlena *) DatumGetPointer(d), 0, + DICT_SAMPLE_LIMIT); + len = VARSIZE_ANY_EXHDR(v); + if (len >= 8) + { + if ((ntrain + nheld) % 5 == 4) + { + appendBinaryStringInfo(&held, VARDATA_ANY(v), len); + heldsizes[nheld++] = len; + } + else + { + appendBinaryStringInfo(&train, VARDATA_ANY(v), len); + trainsizes[ntrain++] = len; + samplebytes += len; + } + } + pfree(v); + if (ntrain + nheld >= DICT_MAX_SAMPLES || + samplebytes >= DICT_EVOLVE_SAMPLE_BYTES) + break; + } + + /* + * Too few values to train on. With no current dictionary there are + * no gates to restart, so the next ANALYZE would sample again; that + * costs only the sampling, since nothing is trained. + */ + if (ntrain < DICT_EVOLVE_MIN_SAMPLES) + goto declined; + + /* + * (e) size the candidate. zstd wants about DICT_SAMPLE_FACTOR times + * the dictionary in samples, so shrink to fit a small sample; but cap + * at the usual default size, because a much larger dictionary trained + * on a block-clustered ANALYZE sample tends to overfit and generalize + * worse on held-out data, not better. Train it on the 80%. + */ + dictsize = (int32) (samplebytes / DICT_SAMPLE_FACTOR); + if (dictsize > DICT_DEFAULT_SIZE) + dictsize = DICT_DEFAULT_SIZE; + if (dictsize < DICT_MIN_SIZE) + dictsize = DICT_MIN_SIZE; + candbuf = palloc(dictsize); + candlen = routine->train(train.data, trainsizes, ntrain, dictsize, + candbuf); + if (candlen == 0) + goto declined; + + /* + * (f) held-out test. With no current dictionary, adopt the first one + * unconditionally (its baseline is plain zstd, already accepted by + * the manual path). Otherwise compare total compressed held-out + * bytes and adopt only on a margin win. + */ + if (!OidIsValid(curid)) + adopt = true; + else if (nheld == 0) + adopt = false; /* nothing to judge by; keep current */ + else + { + size_t curtotal; + size_t candtotal; + + drel = table_open(ToastDictionaryRelationId, AccessShareLock); + (void) dict_current_info(drel, RelationGetRelid(onerel), attnum, + &currows, &curtime, &curbytes, true); + table_close(drel, AccessShareLock); + + /* one call per dictionary: each is digested once, not per value */ + curtotal = routine->compressed_total(VARDATA_ANY(curbytes), + VARSIZE_ANY_EXHDR(curbytes), + held.data, heldsizes, nheld); + candtotal = routine->compressed_total(candbuf, candlen, + held.data, heldsizes, nheld); + + adopt = curtotal != 0 && candtotal != 0 && + (double) candtotal <= (double) curtotal * + (1.0 - dictionary_evolution_min_gain); + + elog(DEBUG1, "dictionary evolution for \"%s\".\"%s\": heldout=%d dictsize=%d cur=%zu cand=%zu gain=%.1f%% adopt=%d", + RelationGetRelationName(onerel), NameStr(att->attname), + nheld, dictsize, curtotal, candtotal, + curtotal ? 100.0 * (1.0 - (double) candtotal / curtotal) : 0.0, + adopt); + } + + /* (g) adopt on a win */ + if (adopt) + { + dict_install(onerel, attnum, InvalidOid, true, + (int64) totalrows, GetCurrentTimestamp(), + candbuf, candlen); + goto done; + } + +declined: + + /* + * Nothing was adopted. Restart the growth and cooldown gates anyway, + * as if the current dictionary had just been trained, so that the + * following ANALYZEs do not sample and train again on the same data. + */ + if (OidIsValid(curid)) + dict_set_gates(onerel, attnum, (int64) totalrows, + GetCurrentTimestamp()); + +done: + ReleaseCurrentSubTransaction(); + MemoryContextSwitchTo(oldcxt); + CurrentResourceOwner = oldowner; + } + PG_CATCH(); + { + ErrorData *edata; + + MemoryContextSwitchTo(oldcxt); + edata = CopyErrorData(); + FlushErrorState(); + RollbackAndReleaseCurrentSubTransaction(); + MemoryContextSwitchTo(oldcxt); + CurrentResourceOwner = oldowner; + MemoryContextDelete(evolvecxt); + + /* + * A cancel (including statement_timeout) must stop ANALYZE as it + * would anywhere else. ProcessInterrupts() clears QueryCancelPending + * before raising it, so test the error code; also re-throw if an + * interrupt is still pending. + */ + if (edata->sqlerrcode == ERRCODE_QUERY_CANCELED || + QueryCancelPending || ProcDiePending) + ReThrowError(edata); + + ereport(WARNING, + (errcode(edata->sqlerrcode), + errmsg("automatic dictionary evolution for column \"%s\" of relation \"%s.%s\" was skipped", + NameStr(att->attname), + get_namespace_name(RelationGetNamespace(onerel)), + RelationGetRelationName(onerel)), + errdetail_internal("%s", edata->message))); + FreeErrorData(edata); + return; + } + PG_END_TRY(); + + MemoryContextDelete(evolvecxt); +} + /* * binary_upgrade_add_toast_dictionary(rel regclass, attnum int2, dictid oid, - * current bool, dict bytea) + * current bool, trainrows int8, + * traintime int8, dict bytea) * * pg_upgrade copies the heap files as they are, and with them the dictionary * OIDs embedded in each datum, so dictionaries are restored with their OIDs. + * The evolution bookkeeping (trainrows, traintime) is carried too, so the + * growth and cooldown gates are unchanged across an upgrade. */ Datum binary_upgrade_add_toast_dictionary(PG_FUNCTION_ARGS) @@ -1163,7 +1788,9 @@ binary_upgrade_add_toast_dictionary(PG_FUNCTION_ARGS) AttrNumber attnum = PG_GETARG_INT16(1); Oid dictid = PG_GETARG_OID(2); bool current = PG_GETARG_BOOL(3); - bytea *dict = PG_GETARG_BYTEA_PP(4); + int64 trainrows = PG_GETARG_INT64(4); + TimestampTz traintime = (TimestampTz) PG_GETARG_INT64(5); + bytea *dict = PG_GETARG_BYTEA_PP(6); Relation rel; if (!IsBinaryUpgrade) @@ -1181,8 +1808,8 @@ binary_upgrade_add_toast_dictionary(PG_FUNCTION_ARGS) if (attnum <= 0 || attnum > RelationGetNumberOfAttributes(rel)) elog(ERROR, "invalid attribute number %d for relation %u", attnum, relid); - dict_install(rel, attnum, dictid, current, VARDATA_ANY(dict), - VARSIZE_ANY_EXHDR(dict)); + dict_install(rel, attnum, dictid, current, trainrows, traintime, + VARDATA_ANY(dict), VARSIZE_ANY_EXHDR(dict)); table_close(rel, NoLock); PG_RETURN_VOID(); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 0c714e21a4e..0a4e78a22a8 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3954,7 +3954,7 @@ heap_truncate(List *relids) heap_truncate_one_rel(rel); /* No value references its old compression dictionaries any more */ - toast_dictionary_retire(RelationGetRelid(rel)); + toast_dictionary_retire(RelationGetRelid(rel), true); /* Close the relation, but keep exclusive lock on it until commit */ table_close(rel, NoLock); diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index 094fbedc307..870f71705d7 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -288,7 +288,11 @@ CREATE VIEW pg_toast_dictionaries WITH (security_barrier) AS c.relname AS tablename, a.attname AS attname, d.tdcurrent AS is_current, - octet_length(d.tddict) AS dict_size + octet_length(d.tddict) AS dict_size, + d.tdtrainrows AS trained_rows, + CASE WHEN d.tdtraintime <> 0 THEN + to_timestamp(946684800 + d.tdtraintime / 1000000.0) + END AS trained_at FROM pg_toast_dictionary d JOIN pg_class c ON (c.oid = d.tdrelid) JOIN pg_attribute a ON (c.oid = a.attrelid AND a.attnum = d.tdattnum) LEFT JOIN pg_namespace n ON (n.oid = c.relnamespace) diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c index 1857f8a484d..52533bf3d38 100644 --- a/src/backend/commands/analyze.c +++ b/src/backend/commands/analyze.c @@ -22,6 +22,8 @@ #include "access/relation.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" +#include "access/toast_dictionary.h" #include "access/transam.h" #include "access/tupconvert.h" #include "access/visibilitymap.h" @@ -613,6 +615,25 @@ do_analyze_rel(Relation onerel, const VacuumParams *params, stats->stadistinct = n_distinct; } + /* + * Automatic TOAST dictionary evolution: for an opted-in column + * that uses a dictionary compression method, consider retraining + * from the sample rows we already fetched. A quick + * attcompression check keeps this free for every other column. + * Only on the table's own rows, not inherited stats. + */ + if (!inh && stats->tupattnum > 0 && + stats->tupattnum <= onerel->rd_att->natts) + { + char cmethod = TupleDescAttr(onerel->rd_att, + stats->tupattnum - 1)->attcompression; + + if (cmethod == TOAST_ZSTD_DICT_COMPRESSION || + cmethod == TOAST_LZ4_DICT_COMPRESSION) + toast_dictionary_maybe_evolve(onerel, stats->tupattnum, + rows, numrows, totalrows); + } + MemoryContextReset(col_context); } diff --git a/src/backend/commands/repack.c b/src/backend/commands/repack.c index a567fe4f040..6724b9092d5 100644 --- a/src/backend/commands/repack.c +++ b/src/backend/commands/repack.c @@ -2159,7 +2159,7 @@ finish_heap_swap(Oid OIDOldHeap, Oid OIDNewHeap, * that used a non-current compression dictionary, so those dictionaries * are no longer referenced and can go. */ - toast_dictionary_retire(OIDOldHeap); + toast_dictionary_retire(OIDOldHeap, false); /* Destroy new heap with old filenumber */ object.classId = RelationRelationId; diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 37e9d317e8f..21bb2452728 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -2298,7 +2298,7 @@ ExecuteTruncateGuts(List *explicit_rels, * No value references a compression dictionary of this relation's * columns any more, so retire the ones a rewrite would. */ - toast_dictionary_retire(RelationGetRelid(rel)); + toast_dictionary_retire(RelationGetRelid(rel), true); pgstat_count_truncate(rel); } diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat index 75d832eb570..73672db68eb 100644 --- a/src/backend/utils/misc/guc_parameters.dat +++ b/src/backend/utils/misc/guc_parameters.dat @@ -820,6 +820,40 @@ check_hook => 'check_default_with_oids', }, +{ name => 'dictionary_evolution_autovacuum', type => 'bool', context => 'PGC_SIGHUP', group => 'VACUUM_DEFAULT', + short_desc => 'Allows autovacuum\'s ANALYZE to retrain TOAST compression dictionaries.', + long_desc => 'When off, automatic dictionary evolution runs only in a manual ANALYZE.', + variable => 'dictionary_evolution_autovacuum', + boot_val => 'true', +}, + +{ name => 'dictionary_evolution_max_retained', type => 'int', context => 'PGC_SUSET', group => 'VACUUM_DEFAULT', + short_desc => 'Maximum number of TOAST compression dictionaries retained per column by automatic evolution.', + long_desc => 'When a column reaches this many dictionaries, automatic evolution stops retraining it until a table rewrite (VACUUM FULL / CLUSTER / REPACK) retires the old ones.', + variable => 'dictionary_evolution_max_retained', + boot_val => '8', + min => '1', + max => 'INT_MAX', +}, + +{ name => 'dictionary_evolution_min_gain', type => 'real', context => 'PGC_SUSET', group => 'VACUUM_DEFAULT', + short_desc => 'Minimum fractional size reduction a candidate TOAST compression dictionary must achieve to be adopted.', + long_desc => 'A candidate dictionary is adopted only if it compresses a held-out sample to at most (1 - this) times the size the current dictionary achieves.', + variable => 'dictionary_evolution_min_gain', + boot_val => '0.05', + min => '0.0', + max => '1.0', +}, + +{ name => 'dictionary_evolution_min_interval', type => 'int', context => 'PGC_SUSET', group => 'VACUUM_DEFAULT', + short_desc => 'Minimum time between automatic retrainings of a column\'s TOAST compression dictionary.', + flags => 'GUC_UNIT_S', + variable => 'dictionary_evolution_min_interval', + boot_val => '86400', + min => '0', + max => 'INT_MAX', +}, + { name => 'dynamic_library_path', type => 'string', context => 'PGC_SUSET', group => 'CLIENT_CONN_OTHER', short_desc => 'Sets the path for dynamically loadable modules.', long_desc => 'If a dynamically loadable module needs to be opened and the specified name does not have a directory component (i.e., the name does not contain a slash), the system will search this path for the specified file.', diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c index 7802557b5f4..4f4c6dc95e3 100644 --- a/src/backend/utils/misc/guc_tables.c +++ b/src/backend/utils/misc/guc_tables.c @@ -34,6 +34,7 @@ #include "access/htup_details.h" #include "access/slru.h" #include "access/toast_compression.h" +#include "access/toast_dictionary.h" #include "access/twophase.h" #include "access/xlog_internal.h" #include "access/xlogprefetcher.h" diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index c4a61f4d7fb..ddc516d8da9 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -775,6 +775,15 @@ # - Default Behavior - #vacuum_truncate = on # enable truncation after vacuum +#dictionary_evolution_autovacuum = on # let autovacuum retrain TOAST + # compression dictionaries +#dictionary_evolution_max_retained = 8 # max TOAST dictionaries kept per + # column by automatic evolution +#dictionary_evolution_min_gain = 0.05 # min fractional size win to adopt a + # retrained TOAST dictionary +#dictionary_evolution_min_interval = 86400s + # min time between automatic TOAST + # dictionary retrainings # - Freezing - diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 5c520e9e32d..9b0ce0b8cfa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -5880,7 +5880,8 @@ binary_upgrade_dump_toast_dictionaries(Archive *fout, PQExpBuffer q, query = createPQExpBuffer(); appendPQExpBuffer(query, - "SELECT d.oid, d.tdattnum, d.tdcurrent, d.tddict " + "SELECT d.oid, d.tdattnum, d.tdcurrent, d.tdtrainrows, " + "d.tdtraintime, d.tddict " "FROM pg_catalog.pg_toast_dictionary d " "WHERE d.tdrelid = '%u'::pg_catalog.oid " "ORDER BY d.oid", @@ -5896,7 +5897,10 @@ binary_upgrade_dump_toast_dictionaries(Archive *fout, PQExpBuffer q, PQgetvalue(res, i, 1), PQgetvalue(res, i, 0), PQgetvalue(res, i, 2)[0] == 't' ? "true" : "false"); - appendStringLiteralAH(q, PQgetvalue(res, i, 3), fout); + appendPQExpBuffer(q, "%s::pg_catalog.int8, %s::pg_catalog.int8, ", + PQgetvalue(res, i, 3), + PQgetvalue(res, i, 4)); + appendStringLiteralAH(q, PQgetvalue(res, i, 5), fout); appendPQExpBufferStr(q, "::pg_catalog.bytea);\n"); } diff --git a/src/bin/psql/tab-complete.in.c b/src/bin/psql/tab-complete.in.c index 24f3c65f2df..5d96c317938 100644 --- a/src/bin/psql/tab-complete.in.c +++ b/src/bin/psql/tab-complete.in.c @@ -2968,7 +2968,9 @@ match_previous_words(int pattern_id, /* ALTER TABLE ALTER [COLUMN] SET ( */ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") || Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "(")) - COMPLETE_WITH("n_distinct", "n_distinct_inherited"); + COMPLETE_WITH("dictionary_evolution", "dictionary_evolution_growth", + "dictionary_evolution_min_rows", + "n_distinct", "n_distinct_inherited"); /* ALTER TABLE ALTER [COLUMN] SET COMPRESSION */ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "COMPRESSION") || Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "COMPRESSION")) diff --git a/src/include/access/toast_dictionary.h b/src/include/access/toast_dictionary.h index 41b4fdf5c85..e2ec273591a 100644 --- a/src/include/access/toast_dictionary.h +++ b/src/include/access/toast_dictionary.h @@ -12,6 +12,7 @@ #ifndef TOAST_DICTIONARY_H #define TOAST_DICTIONARY_H +#include "access/htup.h" #include "utils/relcache.h" #include "varatt.h" @@ -30,16 +31,33 @@ /* * Per-codec dictionary routines. The catalog, lifecycle, training, and the * datum layout above are codec-independent and live in toast_dictionary.c; - * only these three calls differ per codec. compress() lays out the header - * and writes the codec payload after VARHDRSZ_COMPRESSED_DICT; it returns - * NULL if the value did not get smaller. decompress() and - * decompress_slice() read a datum this codec produced. + * only these calls differ per codec. compress() lays out the header and + * writes the codec payload after VARHDRSZ_COMPRESSED_DICT; it returns NULL if + * the value did not get smaller. decompress() and decompress_slice() read a + * datum this codec produced. + * + * train() and compressed_total() exist only for automatic dictionary + * evolution (see toast_dictionary_maybe_evolve()). train() builds a + * dictionary of at most max_dict_bytes from nsamples concatenated samples + * (sizes[] gives each one's length), writing it into dictbuf and returning + * its length, or 0 on failure (a decline, not an error). compressed_total() + * compresses each of nvals concatenated values (sizes[] gives each one's + * length) against raw dictionary bytes and returns the total compressed byte + * count without storing anything, or 0 if any value fails, so the evolution + * policy can weigh a candidate dictionary against the current one with one + * call per dictionary. Both are codec contract points only: the policy never + * interprets the dictionary bytes. */ typedef struct DictCompressRoutine { varlena *(*compress) (const varlena *value, Oid dictid); varlena *(*decompress) (const varlena *value); varlena *(*decompress_slice) (const varlena *value, int32 slicelength); + size_t (*train) (const char *samples, const size_t *sizes, + int nsamples, size_t max_dict_bytes, void *dictbuf); + size_t (*compressed_total) (const char *dict, size_t dictlen, + const char *vals, const size_t *sizes, + int nvals); } DictCompressRoutine; /* Current dictionary of rel.attnum, or InvalidOid if none */ @@ -60,6 +78,23 @@ extern varlena *toast_dict_decompress_datum_slice(const varlena *value, int32 slicelength); extern void RemoveToastDictionaries(Oid relid, AttrNumber attnum); -extern void toast_dictionary_retire(Oid relid); +extern void toast_dictionary_retire(Oid relid, bool truncated); + +/* + * Automatic dictionary evolution, called from ANALYZE for opted-in columns + * that use a dictionary method. Reuses ANALYZE's already-fetched sample + * rows; trains a candidate, runs a held-out size test, and installs it only + * on a margin win, all in the maintenance process. A no-op for every other + * column. + */ +extern void toast_dictionary_maybe_evolve(Relation onerel, AttrNumber attnum, + HeapTuple *rows, int numrows, + double totalrows); + +/* GUCs controlling automatic dictionary evolution; see toast_dictionary.c */ +extern PGDLLIMPORT double dictionary_evolution_min_gain; +extern PGDLLIMPORT int dictionary_evolution_min_interval; +extern PGDLLIMPORT int dictionary_evolution_max_retained; +extern PGDLLIMPORT bool dictionary_evolution_autovacuum; #endif /* TOAST_DICTIONARY_H */ diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 9a75ab158c0..57af8efea76 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -12032,7 +12032,7 @@ { oid => '9266', descr => 'for use by pg_upgrade', proname => 'binary_upgrade_add_toast_dictionary', provolatile => 'v', proparallel => 'u', prorettype => 'void', - proargtypes => 'regclass int2 oid bool bytea', + proargtypes => 'regclass int2 oid bool int8 int8 bytea', prosrc => 'binary_upgrade_add_toast_dictionary' }, { oid => '4545', descr => 'for use by pg_upgrade', proname => 'binary_upgrade_set_next_heap_relfilenode', provolatile => 'v', diff --git a/src/include/catalog/pg_toast_dictionary.h b/src/include/catalog/pg_toast_dictionary.h index a2abc91ae9c..b01a92090b8 100644 --- a/src/include/catalog/pg_toast_dictionary.h +++ b/src/include/catalog/pg_toast_dictionary.h @@ -40,9 +40,13 @@ CATALOG(pg_toast_dictionary,9260,ToastDictionaryRelationId) Oid tdrelid BKI_LOOKUP(pg_class); /* owning relation */ int16 tdattnum; /* owning column */ bool tdcurrent; /* used to compress new values? */ + int64 tdtrainrows; /* reltuples estimate when trained, for the + * auto-evolution growth trigger */ + int64 tdtraintime; /* training time as a TimestampTz, for the + * auto-evolution cooldown */ #ifdef CATALOG_VARLEN /* variable-length fields start here */ - bytea tddict BKI_FORCE_NOT_NULL; /* zstd dictionary content */ + bytea tddict BKI_FORCE_NOT_NULL; /* trained dictionary content */ #endif } FormData_pg_toast_dictionary; diff --git a/src/include/utils/attoptcache.h b/src/include/utils/attoptcache.h index 172055fcd10..e0f7f993633 100644 --- a/src/include/utils/attoptcache.h +++ b/src/include/utils/attoptcache.h @@ -21,6 +21,9 @@ typedef struct AttributeOpts int32 vl_len_; /* varlena header (do not touch directly!) */ float8 n_distinct; float8 n_distinct_inherited; + bool dictionary_evolution; + float8 dictionary_evolution_growth; + int32 dictionary_evolution_min_rows; } AttributeOpts; extern AttributeOpts *get_attribute_options(Oid attrelid, int attnum); diff --git a/src/test/regress/expected/compression_dict_evolve.out b/src/test/regress/expected/compression_dict_evolve.out new file mode 100644 index 00000000000..b11056f0616 --- /dev/null +++ b/src/test/regress/expected/compression_dict_evolve.out @@ -0,0 +1,292 @@ +-- Tests for automatic TOAST dictionary evolution during ANALYZE: an opted-in +-- column retrains its dictionary from ANALYZE's sample, adopts a candidate +-- only when a held-out size test shows a margin win, rejects churn when no +-- candidate wins, and stops at the retained-dictionary cap. See +-- toast_dictionary_maybe_evolve(). +SELECT NOT(enumvals @> '{zstd}') AS skip_test FROM pg_settings WHERE + name = 'default_toast_compression' \gset +\if :skip_test + \echo '*** skipping TOAST dictionary evolution tests (not supported) ***' + \quit +\endif +CREATE SCHEMA dict_evolve; +SET search_path TO dict_evolve, public; +-- Lenient gates so the test is deterministic: no cooldown and any growth +-- (some ANALYZEs below run on an unchanged row count, so the growth gate must +-- accept equality), with a low row floor. The 5% margin keeps its default +-- so the held-out size test is the real adoption decision. +SET dictionary_evolution_min_interval = 0; +-- Two visibly different value distributions, each wide enough to be TOASTed +-- and compressed on disk (a few kB), so the test exercises real +-- dictionary compression and not just catalog-row churn. A dictionary +-- trained on one compresses it well and the other poorly, so switching the +-- stored distribution produces a candidate that wins the held-out test. +CREATE FUNCTION docA(i int) RETURNS text LANGUAGE SQL IMMUTABLE AS $$ + SELECT repeat( + format('{"customer_id": %s, "status": "active", "region": "us-east", ' + '"plan": "enterprise", "features": ["sso", "audit", "backup"], ' + '"env": "production", "tier": "gold", "shard": "a-%s"}' || chr(10), + i, i % 7), 40) +$$; +CREATE FUNCTION docB(i int) RETURNS text LANGUAGE SQL IMMUTABLE AS $$ + SELECT repeat( + format('warehousepallet' + 'cold-storage%sin-transit' + 'freightwaysstandard' || chr(10), + i, i % 7), 40) +$$; +CREATE TABLE evo (id int, f1 text COMPRESSION zstd_dict) + WITH (autovacuum_enabled = off); +ALTER TABLE evo ALTER COLUMN f1 SET (dictionary_evolution = on, + dictionary_evolution_growth = 1.0, + dictionary_evolution_min_rows = 50); +SELECT 'evo'::regclass::oid AS evo_oid \gset +-- no dictionary yet +SELECT count(*) AS ndicts FROM pg_toast_dictionary WHERE tdrelid = :evo_oid; + ndicts +-------- + 0 +(1 row) + +-- A fixed set of rows, distribution A. ANALYZE adopts the first dictionary +-- unconditionally once the row floor is cleared. +INSERT INTO evo SELECT g, docA(g) FROM generate_series(1, 4000) g; +ANALYZE evo; +SELECT count(*) AS ndicts, count(*) FILTER (WHERE tdcurrent) AS ncurrent + FROM pg_toast_dictionary WHERE tdrelid = :evo_oid; + ndicts | ncurrent +--------+---------- + 1 | 1 +(1 row) + +SELECT oid AS dicta FROM pg_toast_dictionary + WHERE tdrelid = :evo_oid AND tdcurrent \gset +-- The rows were inserted before there was a dictionary, so they are plain +-- zstd. Rewrite them, so that they are compressed with dictionary A, and +-- confirm they are wide enough to be compressed on disk (stored smaller than +-- raw), so the test exercises real on-disk compression and not just +-- catalog-row churn on sub-threshold values. +UPDATE evo SET f1 = f1 || ''; +SELECT pg_column_compression(f1) AS cm, + bool_and(pg_column_size(f1) < octet_length(f1)) AS compressed_on_disk, + count(*) + FROM evo GROUP BY 1; + cm | compressed_on_disk | count +-----------+--------------------+------- + zstd_dict | t | 4000 +(1 row) + +-- Same distribution again: a re-trained candidate does not beat the current +-- dictionary by the margin, so nothing is adopted. The gates restart all the +-- same, so that the next ANALYZE does not train again on the same data. +SELECT tdtraintime AS traintime_a FROM pg_toast_dictionary + WHERE tdrelid = :evo_oid AND tdcurrent \gset +ANALYZE evo; +SELECT count(*) AS ndicts FROM pg_toast_dictionary WHERE tdrelid = :evo_oid; + ndicts +-------- + 1 +(1 row) + +SELECT oid = :dicta AS still_same_current, tdtraintime > :traintime_a AS gates_restarted + FROM pg_toast_dictionary WHERE tdrelid = :evo_oid AND tdcurrent; + still_same_current | gates_restarted +--------------------+----------------- + t | t +(1 row) + +-- Drift: add a batch of distribution-B rows (the A rows stay, so values +-- compressed with the A dictionary coexist with B rows on disk). ANALYZE +-- should find a candidate that beats the A-trained dictionary on the held-out +-- B-dominated sample and adopt it: the count rises and the current dictionary +-- changes. +INSERT INTO evo SELECT g, docB(g) FROM generate_series(4001, 12000) g; +ANALYZE evo; +SELECT count(*) AS ndicts FROM pg_toast_dictionary WHERE tdrelid = :evo_oid; + ndicts +-------- + 2 +(1 row) + +SELECT oid <> :dicta AS current_changed FROM pg_toast_dictionary + WHERE tdrelid = :evo_oid AND tdcurrent; + current_changed +----------------- + t +(1 row) + +SELECT oid AS dictb FROM pg_toast_dictionary + WHERE tdrelid = :evo_oid AND tdcurrent \gset +-- Both generations read back correctly: the A rows, still compressed with +-- the now non-current dictionary A, and the B rows, which were inserted +-- while A was current and so are compressed with it too. New rows use B. +INSERT INTO evo SELECT g, docB(g) FROM generate_series(12001, 12100) g; +SELECT id <= 4000 AS a_rows, pg_column_compression(f1) AS cm, count(*) + FROM evo GROUP BY 1, 2 ORDER BY 1, 2; + a_rows | cm | count +--------+-----------+------- + f | zstd_dict | 8100 + t | zstd_dict | 4000 +(2 rows) + +SELECT bool_and(f1 = docA(id)) AS a_ok FROM evo WHERE id <= 4000; + a_ok +------ + t +(1 row) + +SELECT bool_and(f1 = docB(id)) AS b_ok FROM evo WHERE id > 4000; + b_ok +------ + t +(1 row) + +SELECT count(*) FILTER (WHERE NOT tdcurrent) AS retired_kept, + count(*) FILTER (WHERE tdcurrent) AS ncurrent + FROM pg_toast_dictionary WHERE tdrelid = :evo_oid; + retired_kept | ncurrent +--------------+---------- + 1 | 1 +(1 row) + +-- A rewrite recompresses every value with B and drops A. +VACUUM FULL evo; +SELECT count(*) AS ndicts_after_rewrite, bool_and(oid = :dictb) AS only_b + FROM pg_toast_dictionary WHERE tdrelid = :evo_oid; + ndicts_after_rewrite | only_b +----------------------+-------- + 1 | t +(1 row) + +SELECT bool_and(f1 = CASE WHEN id <= 4000 THEN docA(id) ELSE docB(id) END) AS all_ok + FROM evo; + all_ok +-------- + t +(1 row) + +-- Stable again on B: ANALYZE must not churn. +ANALYZE evo; +SELECT oid = :dictb AS still_same_current FROM pg_toast_dictionary + WHERE tdrelid = :evo_oid AND tdcurrent; + still_same_current +-------------------- + t +(1 row) + +-- Cap: with a low cap, repeated drift keeps adopting only until the cap is +-- reached, after which evolution declines and the catalog stops growing. The +-- cap message goes to the server log; here we just check the bound holds. +SET dictionary_evolution_max_retained = 3; +DO $$ +DECLARE + i int; +BEGIN + FOR i IN 1..6 LOOP + IF i % 2 = 0 THEN + UPDATE evo SET f1 = docA(id); + ELSE + UPDATE evo SET f1 = docB(id); + END IF; + ANALYZE evo; + END LOOP; +END $$; +SELECT count(*) <= 3 AS within_cap, + count(*) FILTER (WHERE tdcurrent) AS ncurrent + FROM pg_toast_dictionary WHERE tdrelid = :evo_oid; + within_cap | ncurrent +------------+---------- + t | 1 +(1 row) + +SELECT bool_and(f1 = docA(id)) AS all_equal FROM evo; + all_equal +----------- + t +(1 row) + +-- A dictionary trained by hand before the table was ever analyzed records no +-- row count. The first ANALYZE takes its count as the baseline for growth, +-- instead of seeing growth from zero, and keeps the hand-trained dictionary. +-- It does so even while the table is below the row floor. +CREATE TABLE evo2 (id int, f1 text COMPRESSION zstd_dict) + WITH (autovacuum_enabled = off); +ALTER TABLE evo2 ALTER COLUMN f1 SET (dictionary_evolution = on, + dictionary_evolution_min_rows = 5000); +INSERT INTO evo2 SELECT g, docB(g) FROM generate_series(1, 2000) g; +SELECT pg_train_toast_dictionary('evo2', 'f1', 4096) AS dict2 \gset +SELECT tdtrainrows FROM pg_toast_dictionary WHERE oid = :dict2; + tdtrainrows +------------- + 0 +(1 row) + +ANALYZE evo2; +SELECT oid = :dict2 AS kept, tdtrainrows > 0 AS baseline_set + FROM pg_toast_dictionary WHERE tdrelid = 'evo2'::regclass AND tdcurrent; + kept | baseline_set +------+-------------- + t | t +(1 row) + +-- pg_toast_dictionaries shows the training time as a timestamptz +SELECT trained_rows > 0 AS rows_known, + trained_at BETWEEN now() - interval '1 hour' AND now() + interval '1 hour' AS recent + FROM pg_toast_dictionaries + WHERE schemaname = 'dict_evolve' AND tablename = 'evo2' AND is_current; + rows_known | recent +------------+-------- + t | t +(1 row) + +-- TRUNCATE resets the recorded row count, so that the table's new contents, +-- not its size before the truncation, are the baseline for growth. +SELECT tdtrainrows > 0 AS has_baseline FROM pg_toast_dictionary + WHERE tdrelid = 'evo2'::regclass AND tdcurrent; + has_baseline +-------------- + t +(1 row) + +TRUNCATE evo2; +SELECT tdtrainrows AS baseline_after_truncate FROM pg_toast_dictionary + WHERE tdrelid = 'evo2'::regclass AND tdcurrent; + baseline_after_truncate +------------------------- + 0 +(1 row) + +-- Nothing is installed in a read-only transaction. +CREATE TABLE evo3 (id int, f1 text COMPRESSION zstd_dict) + WITH (autovacuum_enabled = off); +ALTER TABLE evo3 ALTER COLUMN f1 SET (dictionary_evolution = on, + dictionary_evolution_min_rows = 50); +INSERT INTO evo3 SELECT g, docA(g) FROM generate_series(1, 500) g; +BEGIN TRANSACTION READ ONLY; +ANALYZE evo3; +COMMIT; +SELECT count(*) AS ndicts_read_only FROM pg_toast_dictionary + WHERE tdrelid = 'evo3'::regclass; + ndicts_read_only +------------------ + 0 +(1 row) + +ANALYZE evo3; +SELECT count(*) AS ndicts_read_write FROM pg_toast_dictionary + WHERE tdrelid = 'evo3'::regclass; + ndicts_read_write +------------------- + 1 +(1 row) + +RESET dictionary_evolution_max_retained; +RESET dictionary_evolution_min_interval; +RESET search_path; +DROP SCHEMA dict_evolve CASCADE; +NOTICE: drop cascades to 5 other objects +DETAIL: drop cascades to function dict_evolve.doca(integer) +drop cascades to function dict_evolve.docb(integer) +drop cascades to table dict_evolve.evo +drop cascades to table dict_evolve.evo2 +drop cascades to table dict_evolve.evo3 diff --git a/src/test/regress/expected/compression_dict_evolve_1.out b/src/test/regress/expected/compression_dict_evolve_1.out new file mode 100644 index 00000000000..e35cd728759 --- /dev/null +++ b/src/test/regress/expected/compression_dict_evolve_1.out @@ -0,0 +1,11 @@ +-- Tests for automatic TOAST dictionary evolution during ANALYZE: an opted-in +-- column retrains its dictionary from ANALYZE's sample, adopts a candidate +-- only when a held-out size test shows a margin win, rejects churn when no +-- candidate wins, and stops at the retained-dictionary cap. See +-- toast_dictionary_maybe_evolve(). +SELECT NOT(enumvals @> '{zstd}') AS skip_test FROM pg_settings WHERE + name = 'default_toast_compression' \gset +\if :skip_test + \echo '*** skipping TOAST dictionary evolution tests (not supported) ***' +*** skipping TOAST dictionary evolution tests (not supported) *** + \quit diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index ff7fdccb7da..788601c6e6e 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -2880,7 +2880,12 @@ pg_toast_dictionaries| SELECT d.oid AS dictid, c.relname AS tablename, a.attname, d.tdcurrent AS is_current, - octet_length(d.tddict) AS dict_size + octet_length(d.tddict) AS dict_size, + d.tdtrainrows AS trained_rows, + CASE + WHEN (d.tdtraintime <> 0) THEN to_timestamp((((946684800)::numeric + ((d.tdtraintime)::numeric / 1000000.0)))::double precision) + ELSE NULL::timestamp with time zone + END AS trained_at FROM (((pg_toast_dictionary d JOIN pg_class c ON ((c.oid = d.tdrelid))) JOIN pg_attribute a ON (((c.oid = a.attrelid) AND (a.attnum = d.tdattnum)))) diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 16286d39a3d..a38a38e1a4d 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -133,7 +133,7 @@ test: partition_join partition_prune reloptions hash_part indexing partition_agg # ---------- # Another group of parallel tests (compression) # ---------- -test: compression compression_lz4 compression_pglz compression_zstd compression_zstd_dict compression_lz4_dict cluster +test: compression compression_lz4 compression_pglz compression_zstd compression_zstd_dict compression_lz4_dict compression_dict_evolve cluster # event_trigger depends on create_am and cannot run concurrently with # any test that runs DDL diff --git a/src/test/regress/sql/compression_dict_evolve.sql b/src/test/regress/sql/compression_dict_evolve.sql new file mode 100644 index 00000000000..8c36724652f --- /dev/null +++ b/src/test/regress/sql/compression_dict_evolve.sql @@ -0,0 +1,188 @@ +-- Tests for automatic TOAST dictionary evolution during ANALYZE: an opted-in +-- column retrains its dictionary from ANALYZE's sample, adopts a candidate +-- only when a held-out size test shows a margin win, rejects churn when no +-- candidate wins, and stops at the retained-dictionary cap. See +-- toast_dictionary_maybe_evolve(). + +SELECT NOT(enumvals @> '{zstd}') AS skip_test FROM pg_settings WHERE + name = 'default_toast_compression' \gset +\if :skip_test + \echo '*** skipping TOAST dictionary evolution tests (not supported) ***' + \quit +\endif + +CREATE SCHEMA dict_evolve; +SET search_path TO dict_evolve, public; + +-- Lenient gates so the test is deterministic: no cooldown and any growth +-- (some ANALYZEs below run on an unchanged row count, so the growth gate must +-- accept equality), with a low row floor. The 5% margin keeps its default +-- so the held-out size test is the real adoption decision. +SET dictionary_evolution_min_interval = 0; + +-- Two visibly different value distributions, each wide enough to be TOASTed +-- and compressed on disk (a few kB), so the test exercises real +-- dictionary compression and not just catalog-row churn. A dictionary +-- trained on one compresses it well and the other poorly, so switching the +-- stored distribution produces a candidate that wins the held-out test. +CREATE FUNCTION docA(i int) RETURNS text LANGUAGE SQL IMMUTABLE AS $$ + SELECT repeat( + format('{"customer_id": %s, "status": "active", "region": "us-east", ' + '"plan": "enterprise", "features": ["sso", "audit", "backup"], ' + '"env": "production", "tier": "gold", "shard": "a-%s"}' || chr(10), + i, i % 7), 40) +$$; +CREATE FUNCTION docB(i int) RETURNS text LANGUAGE SQL IMMUTABLE AS $$ + SELECT repeat( + format('warehousepallet' + 'cold-storage%sin-transit' + 'freightwaysstandard' || chr(10), + i, i % 7), 40) +$$; + +CREATE TABLE evo (id int, f1 text COMPRESSION zstd_dict) + WITH (autovacuum_enabled = off); +ALTER TABLE evo ALTER COLUMN f1 SET (dictionary_evolution = on, + dictionary_evolution_growth = 1.0, + dictionary_evolution_min_rows = 50); +SELECT 'evo'::regclass::oid AS evo_oid \gset + +-- no dictionary yet +SELECT count(*) AS ndicts FROM pg_toast_dictionary WHERE tdrelid = :evo_oid; + +-- A fixed set of rows, distribution A. ANALYZE adopts the first dictionary +-- unconditionally once the row floor is cleared. +INSERT INTO evo SELECT g, docA(g) FROM generate_series(1, 4000) g; +ANALYZE evo; +SELECT count(*) AS ndicts, count(*) FILTER (WHERE tdcurrent) AS ncurrent + FROM pg_toast_dictionary WHERE tdrelid = :evo_oid; +SELECT oid AS dicta FROM pg_toast_dictionary + WHERE tdrelid = :evo_oid AND tdcurrent \gset + +-- The rows were inserted before there was a dictionary, so they are plain +-- zstd. Rewrite them, so that they are compressed with dictionary A, and +-- confirm they are wide enough to be compressed on disk (stored smaller than +-- raw), so the test exercises real on-disk compression and not just +-- catalog-row churn on sub-threshold values. +UPDATE evo SET f1 = f1 || ''; +SELECT pg_column_compression(f1) AS cm, + bool_and(pg_column_size(f1) < octet_length(f1)) AS compressed_on_disk, + count(*) + FROM evo GROUP BY 1; + +-- Same distribution again: a re-trained candidate does not beat the current +-- dictionary by the margin, so nothing is adopted. The gates restart all the +-- same, so that the next ANALYZE does not train again on the same data. +SELECT tdtraintime AS traintime_a FROM pg_toast_dictionary + WHERE tdrelid = :evo_oid AND tdcurrent \gset +ANALYZE evo; +SELECT count(*) AS ndicts FROM pg_toast_dictionary WHERE tdrelid = :evo_oid; +SELECT oid = :dicta AS still_same_current, tdtraintime > :traintime_a AS gates_restarted + FROM pg_toast_dictionary WHERE tdrelid = :evo_oid AND tdcurrent; + +-- Drift: add a batch of distribution-B rows (the A rows stay, so values +-- compressed with the A dictionary coexist with B rows on disk). ANALYZE +-- should find a candidate that beats the A-trained dictionary on the held-out +-- B-dominated sample and adopt it: the count rises and the current dictionary +-- changes. +INSERT INTO evo SELECT g, docB(g) FROM generate_series(4001, 12000) g; +ANALYZE evo; +SELECT count(*) AS ndicts FROM pg_toast_dictionary WHERE tdrelid = :evo_oid; +SELECT oid <> :dicta AS current_changed FROM pg_toast_dictionary + WHERE tdrelid = :evo_oid AND tdcurrent; +SELECT oid AS dictb FROM pg_toast_dictionary + WHERE tdrelid = :evo_oid AND tdcurrent \gset + +-- Both generations read back correctly: the A rows, still compressed with +-- the now non-current dictionary A, and the B rows, which were inserted +-- while A was current and so are compressed with it too. New rows use B. +INSERT INTO evo SELECT g, docB(g) FROM generate_series(12001, 12100) g; +SELECT id <= 4000 AS a_rows, pg_column_compression(f1) AS cm, count(*) + FROM evo GROUP BY 1, 2 ORDER BY 1, 2; +SELECT bool_and(f1 = docA(id)) AS a_ok FROM evo WHERE id <= 4000; +SELECT bool_and(f1 = docB(id)) AS b_ok FROM evo WHERE id > 4000; +SELECT count(*) FILTER (WHERE NOT tdcurrent) AS retired_kept, + count(*) FILTER (WHERE tdcurrent) AS ncurrent + FROM pg_toast_dictionary WHERE tdrelid = :evo_oid; +-- A rewrite recompresses every value with B and drops A. +VACUUM FULL evo; +SELECT count(*) AS ndicts_after_rewrite, bool_and(oid = :dictb) AS only_b + FROM pg_toast_dictionary WHERE tdrelid = :evo_oid; +SELECT bool_and(f1 = CASE WHEN id <= 4000 THEN docA(id) ELSE docB(id) END) AS all_ok + FROM evo; + +-- Stable again on B: ANALYZE must not churn. +ANALYZE evo; +SELECT oid = :dictb AS still_same_current FROM pg_toast_dictionary + WHERE tdrelid = :evo_oid AND tdcurrent; + +-- Cap: with a low cap, repeated drift keeps adopting only until the cap is +-- reached, after which evolution declines and the catalog stops growing. The +-- cap message goes to the server log; here we just check the bound holds. +SET dictionary_evolution_max_retained = 3; +DO $$ +DECLARE + i int; +BEGIN + FOR i IN 1..6 LOOP + IF i % 2 = 0 THEN + UPDATE evo SET f1 = docA(id); + ELSE + UPDATE evo SET f1 = docB(id); + END IF; + ANALYZE evo; + END LOOP; +END $$; +SELECT count(*) <= 3 AS within_cap, + count(*) FILTER (WHERE tdcurrent) AS ncurrent + FROM pg_toast_dictionary WHERE tdrelid = :evo_oid; +SELECT bool_and(f1 = docA(id)) AS all_equal FROM evo; + +-- A dictionary trained by hand before the table was ever analyzed records no +-- row count. The first ANALYZE takes its count as the baseline for growth, +-- instead of seeing growth from zero, and keeps the hand-trained dictionary. +-- It does so even while the table is below the row floor. +CREATE TABLE evo2 (id int, f1 text COMPRESSION zstd_dict) + WITH (autovacuum_enabled = off); +ALTER TABLE evo2 ALTER COLUMN f1 SET (dictionary_evolution = on, + dictionary_evolution_min_rows = 5000); +INSERT INTO evo2 SELECT g, docB(g) FROM generate_series(1, 2000) g; +SELECT pg_train_toast_dictionary('evo2', 'f1', 4096) AS dict2 \gset +SELECT tdtrainrows FROM pg_toast_dictionary WHERE oid = :dict2; +ANALYZE evo2; +SELECT oid = :dict2 AS kept, tdtrainrows > 0 AS baseline_set + FROM pg_toast_dictionary WHERE tdrelid = 'evo2'::regclass AND tdcurrent; + +-- pg_toast_dictionaries shows the training time as a timestamptz +SELECT trained_rows > 0 AS rows_known, + trained_at BETWEEN now() - interval '1 hour' AND now() + interval '1 hour' AS recent + FROM pg_toast_dictionaries + WHERE schemaname = 'dict_evolve' AND tablename = 'evo2' AND is_current; + +-- TRUNCATE resets the recorded row count, so that the table's new contents, +-- not its size before the truncation, are the baseline for growth. +SELECT tdtrainrows > 0 AS has_baseline FROM pg_toast_dictionary + WHERE tdrelid = 'evo2'::regclass AND tdcurrent; +TRUNCATE evo2; +SELECT tdtrainrows AS baseline_after_truncate FROM pg_toast_dictionary + WHERE tdrelid = 'evo2'::regclass AND tdcurrent; + +-- Nothing is installed in a read-only transaction. +CREATE TABLE evo3 (id int, f1 text COMPRESSION zstd_dict) + WITH (autovacuum_enabled = off); +ALTER TABLE evo3 ALTER COLUMN f1 SET (dictionary_evolution = on, + dictionary_evolution_min_rows = 50); +INSERT INTO evo3 SELECT g, docA(g) FROM generate_series(1, 500) g; +BEGIN TRANSACTION READ ONLY; +ANALYZE evo3; +COMMIT; +SELECT count(*) AS ndicts_read_only FROM pg_toast_dictionary + WHERE tdrelid = 'evo3'::regclass; +ANALYZE evo3; +SELECT count(*) AS ndicts_read_write FROM pg_toast_dictionary + WHERE tdrelid = 'evo3'::regclass; + +RESET dictionary_evolution_max_retained; +RESET dictionary_evolution_min_interval; +RESET search_path; +DROP SCHEMA dict_evolve CASCADE; -- 2.50.1