From 633253a105b248efe219975675ff1ffe642f14d4 Mon Sep 17 00:00:00 2001 From: Greg Burd Date: Sat, 24 Oct 2026 15:03:42 -0400 Subject: [PATCH v11 5/5] Add automatic dictionary evolution during ANALYZE A trained dictionary ages: as a column's data drifts, the dictionary that was best for the old data compresses the new data worse. Retraining was manual (pg_train_toast_dictionary). This adds an opt-in policy that lets ANALYZE retrain a column's dictionary when the data has changed enough to justify it, and only when a new dictionary actually compresses better. The policy is codec-agnostic; it rides the existing dictionary lifecycle and the DictCompressRoutine seam, so it serves zstd_dict and lz4_dict alike. None of it runs on the INSERT/UPDATE path. Opt-in per column, like n_distinct, via attribute options: dictionary_evolution (bool, default off) dictionary_evolution_growth (real, default 2.0) dictionary_evolution_min_rows (int, default 10000) and three GUCs: dictionary_evolution_min_gain (real, default 0.05) dictionary_evolution_min_interval (int seconds, default 86400) dictionary_evolution_max_retained (int, default 8) When ANALYZE processes an opted-in dictionary column, after it has the block sample it calls toast_dictionary_maybe_evolve(): - Gates. Do nothing unless the live row count is at least min_rows, has grown by the growth factor since the current dictionary was trained (pg_toast_dictionary.tdtrainrows), and the current dictionary is older than min_interval (tdtraintime). If the column already has max_retained dictionaries, decline with a LOG and suggest a rewrite, since retirement only happens on rewrite. - Candidate. Build samples from ANALYZE's own rows[] (no extra I/O; same prefix and skip rules as the manual trainer), hold out every fifth value, and train a candidate on the rest through the codec's train() routine. The candidate dictionary size is derived from the sample. - Adopt only on a win. Compress the held-out values with the current dictionary and with the candidate through the codec's compressed_size() routine, and adopt the candidate only if it is at least min_gain smaller on data it was not trained on. The first dictionary of a column is adopted unconditionally. Everything runs in the ANALYZE process and is wrapped so any failure is logged and swallowed, never aborting ANALYZE. Two pg_toast_dictionary columns support the gates: tdtrainrows (the reltuples estimate at training time) and tdtraintime. dict_install() writes them and now invalidates the owning relation's relcache entry, so the current-dictionary OID can be cached there: toast_dictionary_current() fills a small per-relation cache instead of probing the catalog for every compressed value, which is the fast-path cost the lifecycle patch left as an XXX. Reviewed-by: Manuel Reyes Bravo Discussion: https://postgr.es/m/CAFAfj_GGY4W81XjtYLvYOhqPtvfZbBNj=YRS6Mq_0SA9AqEwoQ@mail.gmail.com --- doc/src/sgml/config.sgml | 59 +++ doc/src/sgml/ref/alter_table.sgml | 19 + src/backend/access/common/reloptions.c | 35 +- src/backend/access/common/toast_dictionary.c | 449 +++++++++++++++++- src/backend/commands/analyze.c | 21 + src/backend/utils/misc/guc_parameters.dat | 27 ++ src/backend/utils/misc/guc_tables.c | 1 + src/backend/utils/misc/postgresql.conf.sample | 6 + src/include/access/toast_dictionary.h | 39 +- src/include/catalog/pg_toast_dictionary.h | 6 +- src/include/utils/attoptcache.h | 3 + src/include/utils/rel.h | 12 + .../expected/compression_dict_evolve.out | 152 ++++++ .../expected/compression_dict_evolve_1.out | 11 + src/test/regress/parallel_schedule | 2 +- .../regress/sql/compression_dict_evolve.sql | 111 +++++ 16 files changed, 934 insertions(+), 19 deletions(-) create mode 100644 src/test/regress/expected/compression_dict_evolve.out create mode 100644 src/test/regress/expected/compression_dict_evolve_1.out create mode 100644 src/test/regress/sql/compression_dict_evolve.sql diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index c6a990ca17f..7b213ad4109 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -10516,6 +10516,65 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv; + + dictionary_evolution_max_retained (integer) + + dictionary_evolution_max_retained configuration parameter + + + + + The maximum number of TOAST + compression dictionaries kept per column by automatic dictionary + evolution (enabled with the per-column + dictionary_evolution attribute option). When a + column reaches this many dictionaries, evolution stops retraining it + and logs a message; a table rewrite + (VACUUM FULL, CLUSTER, or + REPACK) retires the superseded dictionaries so + evolution can resume. The default is 8. + + + + + + dictionary_evolution_min_gain (floating point) + + dictionary_evolution_min_gain configuration parameter + + + + + The minimum fractional reduction in compressed size a newly trained + candidate TOAST compression + dictionary must achieve, measured on a held-out sample, before + automatic evolution adopts it in place of the current dictionary. + For example, the default of 0.05 requires the + candidate to compress the held-out sample to at most 95% of the size + the current dictionary achieves. A larger value makes evolution more + conservative. + + + + + + dictionary_evolution_min_interval (integer) + + dictionary_evolution_min_interval configuration parameter + + + + + The minimum time that must pass after a column's + TOAST compression dictionary was + trained before automatic evolution will retrain it again, acting as a + cooldown that bounds churn. If this value is specified without units, + it is taken as seconds. The default is one day + (86400 seconds). + + + + temp_tablespaces (string) diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml index cfd7f28e7b7..00db9489fba 100644 --- a/doc/src/sgml/ref/alter_table.sgml +++ b/doc/src/sgml/ref/alter_table.sgml @@ -404,6 +404,25 @@ WITH ( MODULUS numeric_literal, REM PostgreSQL query planner, refer to . + + Three further per-attribute options control automatic + TOAST compression dictionary + evolution, and apply only to columns that use a dictionary compression + method (zstd_dict or lz4_dict). + dictionary_evolution (boolean, default + false) enables, for the column, retraining of its + compression dictionary from the sample gathered by a subsequent + ANALYZE; a newly + trained dictionary is adopted only if it wins a held-out size test by + the margin set by . + dictionary_evolution_growth (floating point, default + 2.0, minimum 1.0) is the factor by + which the column's live-row count must grow since the current dictionary + was trained before retraining is considered. + dictionary_evolution_min_rows (integer, default + 10000) is a floor below which the column is never + retrained. + Changing per-attribute options acquires a SHARE UPDATE EXCLUSIVE lock. diff --git a/src/backend/access/common/reloptions.c b/src/backend/access/common/reloptions.c index ea9a0417909..44b41e91d6d 100644 --- a/src/backend/access/common/reloptions.c +++ b/src/backend/access/common/reloptions.c @@ -153,6 +153,15 @@ static relopt_bool boolRelOpts[] = }, true }, + { + { + "dictionary_evolution", + "Enables automatic retraining of this column's TOAST compression dictionary during ANALYZE.", + RELOPT_KIND_ATTRIBUTE, + ShareUpdateExclusiveLock + }, + false + }, /* list terminator */ {{NULL}} }; @@ -412,6 +421,15 @@ static relopt_int intRelOpts[] = }, -1, 0, 1024 }, + { + { + "dictionary_evolution_min_rows", + "Minimum live-row count before a column's TOAST compression dictionary is eligible for automatic retraining.", + RELOPT_KIND_ATTRIBUTE, + ShareUpdateExclusiveLock + }, + 10000, 0, INT_MAX + }, /* list terminator */ {{NULL}} @@ -501,6 +519,15 @@ static relopt_real realRelOpts[] = }, 0, -1.0, DBL_MAX }, + { + { + "dictionary_evolution_growth", + "Factor by which a column's live-row count must grow since its TOAST compression dictionary was trained before automatic retraining is eligible.", + RELOPT_KIND_ATTRIBUTE, + ShareUpdateExclusiveLock + }, + 2.0, 1.0, DBL_MAX + }, { { "vacuum_cleanup_index_scale_factor", @@ -2418,7 +2445,13 @@ attribute_reloptions(Datum reloptions, bool validate) { static const relopt_parse_elt tab[] = { {"n_distinct", RELOPT_TYPE_REAL, offsetof(AttributeOpts, n_distinct)}, - {"n_distinct_inherited", RELOPT_TYPE_REAL, offsetof(AttributeOpts, n_distinct_inherited)} + {"n_distinct_inherited", RELOPT_TYPE_REAL, offsetof(AttributeOpts, n_distinct_inherited)}, + {"dictionary_evolution", RELOPT_TYPE_BOOL, + offsetof(AttributeOpts, dictionary_evolution)}, + {"dictionary_evolution_growth", RELOPT_TYPE_REAL, + offsetof(AttributeOpts, dictionary_evolution_growth)}, + {"dictionary_evolution_min_rows", RELOPT_TYPE_INT, + offsetof(AttributeOpts, dictionary_evolution_min_rows)} }; return (bytea *) build_reloptions(reloptions, validate, diff --git a/src/backend/access/common/toast_dictionary.c b/src/backend/access/common/toast_dictionary.c index 2b31b54f326..48e9142253d 100644 --- a/src/backend/access/common/toast_dictionary.c +++ b/src/backend/access/common/toast_dictionary.c @@ -71,6 +71,7 @@ #include "lib/stringinfo.h" #include "miscadmin.h" #include "utils/acl.h" +#include "utils/attoptcache.h" #include "utils/builtins.h" #include "utils/fmgroids.h" #include "utils/inval.h" @@ -79,6 +80,7 @@ #include "utils/rel.h" #include "utils/snapmgr.h" #include "utils/syscache.h" +#include "utils/timestamp.h" #define NO_ZSTD_SUPPORT() \ ereport(ERROR, \ @@ -94,11 +96,48 @@ #define DICT_MIN_SIZE 256 #define DICT_MAX_SIZE (1024 * 1024) +#define DICT_DEFAULT_SIZE (64 * 1024) /* pg_train_toast_dictionary default */ #define DICT_MAX_SAMPLES 10000 #define DICT_SAMPLE_LIMIT (128 * 1024) /* zstd recommends samples totalling about 100 times the dictionary size */ #define DICT_SAMPLE_FACTOR 100 +/* + * GUCs for automatic dictionary evolution. Codec-agnostic policy knobs; see + * toast_dictionary_maybe_evolve(). Declared in toast_dictionary.h. + */ +double dictionary_evolution_min_gain = 0.05; +int dictionary_evolution_min_interval = 86400; +int dictionary_evolution_max_retained = 8; + +/* + * Train a dictionary with zstd's ZDICT trainer. Shared by both codecs: + * lz4_dict dictionaries are also trained by ZDICT (lz4 ships no trainer of + * its own), so lz4_dict_routine.train points here too. Returns the trained + * dictionary's length, written into dictbuf (which must hold max_dict_bytes), + * or 0 on failure -- a decline to retrain, not an error. When the server is + * built without zstd, training is unavailable for both codecs and this + * returns 0. + */ +static size_t +dict_train_zdict(const char *samples, const size_t *sizes, int nsamples, + size_t max_dict_bytes, void *dictbuf) +{ +#ifndef USE_ZSTD + return 0; +#else + size_t dictlen; + + if (nsamples == 0) + return 0; + dictlen = ZDICT_trainFromBuffer(dictbuf, max_dict_bytes, samples, sizes, + nsamples); + if (ZDICT_isError(dictlen)) + return 0; + return dictlen; +#endif +} + /* * The relation owning the dictionaries of rel's columns. During a table * rewrite, values are written into a transient heap that stands for the @@ -167,8 +206,13 @@ dict_lookup(Oid relid, AttrNumber attnum, bool current_only, Relation drel, /* * Current dictionary of a column, or InvalidOid. * - * XXX: this is a catalog index probe per compressed value. Caching the OID - * in the relcache entry would avoid it if it ever shows in a profile. + * Called once per compressed value on the write path, so the result is cached + * in the relcache entry: a single-entry cache keyed by attnum, filled lazily + * from the catalog here. dict_install() changes the current dictionary only + * at maintenance time and then issues CacheInvalidateRelcacheByRelid(), which + * rebuilds this relcache entry and so empties the cache; the next call + * repopulates it with the new OID. A single entry suffices for a scan of one + * column (the common case); a second dictionary column just re-probes. */ Oid toast_dictionary_current(Relation rel, AttrNumber attnum) @@ -178,12 +222,18 @@ toast_dictionary_current(Relation rel, AttrNumber attnum) HeapTuple tup; Oid result = InvalidOid; + if (rel->rd_toastdictattnum == attnum) + return rel->rd_toastdictoid; + drel = table_open(ToastDictionaryRelationId, AccessShareLock); tup = dict_lookup(dict_owner_relid(rel), attnum, true, drel, &scan); if (tup) result = ((Form_pg_toast_dictionary) GETSTRUCT(tup))->oid; systable_endscan(scan); table_close(drel, AccessShareLock); + + rel->rd_toastdictoid = result; + rel->rd_toastdictattnum = attnum; return result; } @@ -415,6 +465,42 @@ zstd_dict_decompress_datum_slice(const varlena *value, int32 slicelength) return result; } +/* + * Compressed size of one value against raw dictionary bytes, for the held-out + * test of dictionary evolution. Builds a one-shot CDict from the given bytes + * (not the per-backend digested cache, which keys on OID and a candidate has + * no OID yet) and returns the compressed byte count without storing anything. + * Returns 0 on any zstd failure so the caller can treat it as "no estimate." + */ +static size_t +zstd_dict_compressed_size(const char *dict, size_t dictlen, + const char *val, size_t len) +{ +#ifndef USE_ZSTD + return 0; +#else + ZSTD_CDict *cd; + size_t bound = ZSTD_compressBound(len); + char *out = palloc(bound); + size_t clen; + + if (dict_cctx == NULL && (dict_cctx = ZSTD_createCCtx()) == NULL) + elog(ERROR, "could not create zstd compression context"); + cd = ZSTD_createCDict(dict, dictlen, ZSTD_CLEVEL_DEFAULT); + if (cd == NULL) + { + pfree(out); + return 0; + } + clen = ZSTD_compress_usingCDict(dict_cctx, out, bound, val, len, cd); + ZSTD_freeCDict(cd); + pfree(out); + if (ZSTD_isError(clen)) + return 0; + return clen; +#endif +} + #ifdef USE_LZ4 /* * Per-backend cache of the raw dictionary bytes most recently used, keyed by @@ -558,6 +644,40 @@ lz4_dict_decompress_datum_slice(const varlena *value, int32 slicelength) return result; } +/* + * Compressed size of one value against raw dictionary bytes, for the held-out + * test of dictionary evolution. Loads the bytes into a fresh lz4 stream and + * returns the compressed byte count without storing anything. Returns 0 on + * any lz4 failure. + */ +static size_t +lz4_dict_compressed_size(const char *dict, size_t dictlen, + const char *val, size_t len) +{ +#ifndef USE_LZ4 + return 0; +#else + LZ4_stream_t *stream; + int32 bound = LZ4_compressBound(len); + char *out; + int32 clen; + + if (bound <= 0) + return 0; + stream = LZ4_createStream(); + if (stream == NULL) + return 0; + LZ4_loadDict(stream, dict, dictlen); + out = palloc(bound); + clen = LZ4_compress_fast_continue(stream, val, out, len, bound, 1); + LZ4_freeStream(stream); + pfree(out); + if (clen <= 0) + return 0; + return clen; +#endif +} + /* * The lz4 dictionary codec. See DictCompressRoutine in toast_dictionary.h. */ @@ -565,6 +685,8 @@ static const DictCompressRoutine lz4_dict_routine = { .compress = lz4_dict_compress_datum, .decompress = lz4_dict_decompress_datum, .decompress_slice = lz4_dict_decompress_datum_slice, + .train = dict_train_zdict, + .compressed_size = lz4_dict_compressed_size, }; /* @@ -575,6 +697,8 @@ static const DictCompressRoutine zstd_dict_routine = { .compress = zstd_dict_compress_datum, .decompress = zstd_dict_decompress_datum, .decompress_slice = zstd_dict_decompress_datum_slice, + .train = dict_train_zdict, + .compressed_size = zstd_dict_compressed_size, }; /* @@ -688,10 +812,12 @@ toast_dictionary_retire(Oid relid) /* * Insert a dictionary for rel.attnum and make it current, retiring the * previous current one. dictid may be given (binary upgrade) or chosen here. + * trainrows is the reltuples estimate at training time, recorded for the + * auto-evolution growth trigger; pass 0 when not training from a live count. */ static Oid dict_install(Relation rel, AttrNumber attnum, Oid dictid, bool current, - const void *dict, size_t dictlen) + int64 trainrows, const void *dict, size_t dictlen) { Relation drel; SysScanDesc scan; @@ -728,12 +854,22 @@ dict_install(Relation rel, AttrNumber attnum, Oid dictid, bool current, ObjectIdGetDatum(RelationGetRelid(rel)); values[Anum_pg_toast_dictionary_tdattnum - 1] = Int16GetDatum(attnum); values[Anum_pg_toast_dictionary_tdcurrent - 1] = BoolGetDatum(current); + values[Anum_pg_toast_dictionary_tdtrainrows - 1] = Int64GetDatum(trainrows); + values[Anum_pg_toast_dictionary_tdtraintime - 1] = + TimestampTzGetDatum(GetCurrentTimestamp()); values[Anum_pg_toast_dictionary_tddict - 1] = PointerGetDatum(dictval); tup = heap_form_tuple(RelationGetDescr(drel), values, nulls); CatalogTupleInsert(drel, tup); heap_freetuple(tup); table_close(drel, RowExclusiveLock); + /* + * The current dictionary of this column changed. Invalidate the owning + * relation's relcache entry so backends drop the cached current-dictionary + * OID (toast_dictionary_current()) and pick up the new one. + */ + CacheInvalidateRelcacheByRelid(RelationGetRelid(rel)); + /* let a later call in the same command see this one's changes */ CommandCounterIncrement(); @@ -791,9 +927,9 @@ dict_check_column(Relation rel, const char *colname) * Takes ShareUpdateExclusiveLock, like ANALYZE: DML continues, concurrent * training and DDL wait. * - * XXX: this samples the first DICT_MAX_SAMPLES rows of a seqscan. Training - * from ANALYZE's block sample is the natural follow-up once the lifecycle is - * agreed on. + * Automatic evolution (toast_dictionary_maybe_evolve) trains from ANALYZE's + * block sample instead; this function keeps its own bounded seqscan for the + * explicit, user-requested case. */ Datum pg_train_toast_dictionary(PG_FUNCTION_ARGS) @@ -859,10 +995,10 @@ pg_train_toast_dictionary(PG_FUNCTION_ARGS) ExecDropSingleTupleTableSlot(slot); dictbuf = palloc(dict_size); - dictlen = nsamples == 0 ? 0 : - ZDICT_trainFromBuffer(dictbuf, dict_size, samples.data, sizes, - nsamples); - if (nsamples == 0 || ZDICT_isError(dictlen)) + dictlen = dict_routine_for_method(TupleDescAttr(RelationGetDescr(rel), + attnum - 1)->attcompression) + ->train(samples.data, sizes, nsamples, dict_size, dictbuf); + if (dictlen == 0) { ereport(NOTICE, (errmsg("could not train a compression dictionary for column \"%s\" from %d values", @@ -871,13 +1007,302 @@ pg_train_toast_dictionary(PG_FUNCTION_ARGS) PG_RETURN_NULL(); } - dictid = dict_install(rel, attnum, InvalidOid, true, dictbuf, dictlen); + dictid = dict_install(rel, attnum, InvalidOid, true, + (int64) rel->rd_rel->reltuples, dictbuf, dictlen); table_close(rel, NoLock); PG_RETURN_OID(dictid); #endif } +/* + * Collect the current dictionary row of rel.attnum, if any, returning its OID + * and, when wantbytes, a palloc'd copy of its dictionary bytes (caller frees). + * Also returns tdtrainrows and tdtraintime for the evolution gates. Returns + * InvalidOid when the column has no current dictionary. + */ +static Oid +dict_current_info(Relation drel, Oid relid, AttrNumber attnum, + int64 *trainrows, int64 *traintime, + bytea **dictbytes, bool wantbytes) +{ + SysScanDesc scan; + HeapTuple tup; + Oid result = InvalidOid; + + *trainrows = 0; + *traintime = 0; + if (dictbytes) + *dictbytes = NULL; + tup = dict_lookup(relid, attnum, true, drel, &scan); + if (tup) + { + Form_pg_toast_dictionary d = (Form_pg_toast_dictionary) GETSTRUCT(tup); + + result = d->oid; + *trainrows = d->tdtrainrows; + *traintime = d->tdtraintime; + if (wantbytes) + *dictbytes = DatumGetByteaPCopy(SysCacheGetAttrNotNull(TOASTDICTOID, tup, + Anum_pg_toast_dictionary_tddict)); + } + systable_endscan(scan); + return result; +} + +/* Count dictionaries (current + non-current) for rel.attnum */ +static int +dict_count(Relation drel, Oid relid, AttrNumber attnum) +{ + SysScanDesc scan; + HeapTuple tup; + int n = 0; + + tup = dict_lookup(relid, attnum, false, drel, &scan); + for (; HeapTupleIsValid(tup); tup = systable_getnext(scan)) + n++; + systable_endscan(scan); + return n; +} + +/* + * toast_dictionary_maybe_evolve + * + * Automatic, best-effort dictionary retraining during ANALYZE. Runs only in + * the maintenance process (ANALYZE holds ShareUpdateExclusiveLock, like the + * manual trainer), reuses ANALYZE's already-fetched sample rows, and installs + * a new current dictionary only when a held-out size test says a candidate + * beats the current one by the configured margin. The write path is never + * involved. Any error is logged and swallowed so it cannot abort ANALYZE. + */ +void +toast_dictionary_maybe_evolve(Relation onerel, AttrNumber attnum, + HeapTuple *rows, int numrows, double totalrows) +{ + AttributeOpts *aopt; + Form_pg_attribute att; + const DictCompressRoutine *routine; + MemoryContext evolvecxt, + oldcxt; + TupleDesc tupdesc = RelationGetDescr(onerel); + + /* (a) opted in, and still a dictionary column? */ + if (attnum <= 0 || attnum > tupdesc->natts) + return; + att = TupleDescAttr(tupdesc, attnum - 1); + if (att->attisdropped || !dict_method(att->attcompression)) + return; + aopt = get_attribute_options(RelationGetRelid(onerel), attnum); + if (aopt == NULL || !aopt->dictionary_evolution) + return; + + routine = dict_routine_for_method(att->attcompression); + + /* + * Everything below runs in a short-lived context so that samples, the + * candidate dictionary, and fetched current-dictionary bytes are all + * freed on return, and in a PG_TRY so a codec or catalog error degrades + * to "no evolution this round" rather than aborting ANALYZE. + */ + evolvecxt = AllocSetContextCreate(CurrentMemoryContext, + "Dictionary evolution", + ALLOCSET_DEFAULT_SIZES); + oldcxt = MemoryContextSwitchTo(evolvecxt); + + PG_TRY(); + { + Relation drel; + Oid curid; + int64 currows; + int64 curtime; + bytea *curbytes = NULL; + int ndicts; + StringInfoData train; + StringInfoData held; + size_t *trainsizes; + size_t *heldsizes; + int ntrain = 0; + int nheld = 0; + int i; + size_t samplebytes = 0; + int32 dictsize; + void *candbuf; + size_t candlen; + bool adopt; + + drel = table_open(ToastDictionaryRelationId, AccessShareLock); + curid = dict_current_info(drel, RelationGetRelid(onerel), attnum, + &currows, &curtime, &curbytes, true); + ndicts = dict_count(drel, RelationGetRelid(onerel), attnum); + table_close(drel, AccessShareLock); + + /* (b) growth, floor, and cooldown gates */ + if (totalrows < (double) aopt->dictionary_evolution_min_rows) + goto done; + if (OidIsValid(curid)) + { + if (totalrows < (double) currows * aopt->dictionary_evolution_growth) + goto done; + if (curtime != 0) + { + /* cooldown in ms, saturated to avoid int overflow */ + int cooldown_ms = dictionary_evolution_min_interval > INT_MAX / 1000 ? + INT_MAX : dictionary_evolution_min_interval * 1000; + + if (!TimestampDifferenceExceeds((TimestampTz) curtime, + GetCurrentTimestamp(), + cooldown_ms)) + goto done; + } + } + + /* (c) cap gate */ + if (ndicts >= dictionary_evolution_max_retained) + { + ereport(LOG, + (errmsg("column \"%s\" of \"%s\" reached the retained-dictionary cap of %d; run VACUUM FULL / REPACK to retire old dictionaries", + NameStr(att->attname), + RelationGetRelationName(onerel), + dictionary_evolution_max_retained))); + goto done; + } + + /* + * (d) build samples from the ANALYZE rows[]. Split deterministically + * into an 80% training set and a 20% held-out set (every 5th sample is + * held out), using the same predicates as the manual trainer: skip + * nulls and values under 8 bytes, detoast a 128kB prefix. + */ + initStringInfo(&train); + initStringInfo(&held); + trainsizes = palloc_array(size_t, numrows); + heldsizes = palloc_array(size_t, numrows); + for (i = 0; i < numrows; i++) + { + bool isnull; + Datum d; + varlena *v; + size_t len; + + CHECK_FOR_INTERRUPTS(); + d = heap_getattr(rows[i], attnum, tupdesc, &isnull); + if (isnull) + continue; + v = detoast_attr_slice((varlena *) DatumGetPointer(d), 0, + DICT_SAMPLE_LIMIT); + len = VARSIZE_ANY_EXHDR(v); + if (len >= 8) + { + if ((ntrain + nheld) % 5 == 4) + { + appendBinaryStringInfo(&held, VARDATA_ANY(v), len); + heldsizes[nheld++] = len; + } + else + { + appendBinaryStringInfo(&train, VARDATA_ANY(v), len); + trainsizes[ntrain++] = len; + samplebytes += len; + } + } + pfree(v); + } + + if (ntrain == 0) + goto done; + + /* + * (e) size the candidate. zstd wants about DICT_SAMPLE_FACTOR times + * the dictionary in samples, so shrink to fit a small sample; but cap + * at the usual default size, because a much larger dictionary trained + * on a block-clustered ANALYZE sample tends to overfit and generalize + * worse on held-out data, not better. Train it on the 80%. + */ + dictsize = (int32) (samplebytes / DICT_SAMPLE_FACTOR); + if (dictsize > DICT_DEFAULT_SIZE) + dictsize = DICT_DEFAULT_SIZE; + if (dictsize < DICT_MIN_SIZE) + dictsize = DICT_MIN_SIZE; + candbuf = palloc(dictsize); + candlen = routine->train(train.data, trainsizes, ntrain, dictsize, + candbuf); + if (candlen == 0) + goto done; + + /* + * (f) held-out test. With no current dictionary, adopt the first one + * unconditionally (its baseline is plain zstd, already accepted by the + * manual path). Otherwise compare total compressed held-out bytes and + * adopt only on a margin win. + */ + if (!OidIsValid(curid)) + adopt = true; + else if (nheld == 0) + adopt = false; /* nothing to judge by; keep current */ + else + { + size_t curtotal = 0; + size_t candtotal = 0; + const char *p = held.data; + bool ok = true; + + for (i = 0; i < nheld; i++) + { + size_t c = routine->compressed_size(VARDATA_ANY(curbytes), + VARSIZE_ANY_EXHDR(curbytes), + p, heldsizes[i]); + size_t n = routine->compressed_size(candbuf, candlen, + p, heldsizes[i]); + + if (c == 0 || n == 0) + { + ok = false; + break; + } + curtotal += c; + candtotal += n; + p += heldsizes[i]; + } + adopt = ok && + (double) candtotal <= (double) curtotal * + (1.0 - dictionary_evolution_min_gain); + + elog(DEBUG1, "dictionary evolution for \"%s\": heldout=%d dictsize=%d cur=%zu cand=%zu gain=%.1f%% adopt=%d", + NameStr(att->attname), nheld, dictsize, curtotal, candtotal, + curtotal ? 100.0 * (1.0 - (double) candtotal / curtotal) : 0.0, + adopt); + } + + /* (g) adopt on a win; otherwise change nothing */ + if (adopt) + dict_install(onerel, attnum, InvalidOid, true, + (int64) totalrows, candbuf, candlen); + +done: + ; + } + PG_CATCH(); + { + ErrorData *edata; + + MemoryContextSwitchTo(oldcxt); + edata = CopyErrorData(); + FlushErrorState(); + ereport(LOG, + (errmsg("automatic dictionary evolution for column \"%s\" of \"%s\" was skipped: %s", + NameStr(att->attname), + RelationGetRelationName(onerel), + edata->message))); + FreeErrorData(edata); + MemoryContextDelete(evolvecxt); + return; + } + PG_END_TRY(); + + MemoryContextSwitchTo(oldcxt); + MemoryContextDelete(evolvecxt); +} + /* * binary_upgrade_add_toast_dictionary(rel regclass, attnum int2, dictid oid, * current bool, dict bytea) @@ -910,7 +1335,7 @@ binary_upgrade_add_toast_dictionary(PG_FUNCTION_ARGS) if (attnum <= 0 || attnum > RelationGetNumberOfAttributes(rel)) elog(ERROR, "invalid attribute number %d for relation %u", attnum, relid); - dict_install(rel, attnum, dictid, current, VARDATA_ANY(dict), + dict_install(rel, attnum, dictid, current, 0, VARDATA_ANY(dict), VARSIZE_ANY_EXHDR(dict)); table_close(rel, NoLock); diff --git a/src/backend/commands/analyze.c b/src/backend/commands/analyze.c index 518d7526255..42ebbedeb87 100644 --- a/src/backend/commands/analyze.c +++ b/src/backend/commands/analyze.c @@ -22,6 +22,8 @@ #include "access/relation.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" +#include "access/toast_dictionary.h" #include "access/transam.h" #include "access/tupconvert.h" #include "access/visibilitymap.h" @@ -613,6 +615,25 @@ do_analyze_rel(Relation onerel, const VacuumParams *params, stats->stadistinct = n_distinct; } + /* + * Automatic TOAST dictionary evolution: for an opted-in column + * that uses a dictionary compression method, consider retraining + * from the sample rows we already fetched. A quick attcompression + * check keeps this free for every other column. Only on the + * table's own rows, not inherited stats. + */ + if (!inh && stats->tupattnum > 0 && + stats->tupattnum <= onerel->rd_att->natts) + { + char cmethod = TupleDescAttr(onerel->rd_att, + stats->tupattnum - 1)->attcompression; + + if (cmethod == TOAST_ZSTD_DICT_COMPRESSION || + cmethod == TOAST_LZ4_DICT_COMPRESSION) + toast_dictionary_maybe_evolve(onerel, stats->tupattnum, + rows, numrows, totalrows); + } + MemoryContextReset(col_context); } diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat index 380679a76c0..5d9da797ba0 100644 --- a/src/backend/utils/misc/guc_parameters.dat +++ b/src/backend/utils/misc/guc_parameters.dat @@ -811,6 +811,33 @@ check_hook => 'check_default_with_oids', }, +{ name => 'dictionary_evolution_max_retained', type => 'int', context => 'PGC_USERSET', group => 'CLIENT_CONN_STATEMENT', + short_desc => 'Maximum number of TOAST compression dictionaries retained per column by automatic evolution.', + long_desc => 'When a column reaches this many dictionaries, automatic evolution stops retraining it until a table rewrite (VACUUM FULL / CLUSTER / REPACK) retires the old ones.', + variable => 'dictionary_evolution_max_retained', + boot_val => '8', + min => '1', + max => 'INT_MAX', +}, + +{ name => 'dictionary_evolution_min_gain', type => 'real', context => 'PGC_USERSET', group => 'CLIENT_CONN_STATEMENT', + short_desc => 'Minimum fractional size reduction a candidate TOAST compression dictionary must achieve to be adopted.', + long_desc => 'A candidate dictionary is adopted only if it compresses a held-out sample to at most (1 - this) times the size the current dictionary achieves.', + variable => 'dictionary_evolution_min_gain', + boot_val => '0.05', + min => '0.0', + max => '1.0', +}, + +{ name => 'dictionary_evolution_min_interval', type => 'int', context => 'PGC_USERSET', group => 'CLIENT_CONN_STATEMENT', + short_desc => 'Minimum time between automatic retrainings of a column\'s TOAST compression dictionary.', + flags => 'GUC_UNIT_S', + variable => 'dictionary_evolution_min_interval', + boot_val => '86400', + min => '0', + max => 'INT_MAX', +}, + { name => 'dynamic_library_path', type => 'string', context => 'PGC_SUSET', group => 'CLIENT_CONN_OTHER', short_desc => 'Sets the path for dynamically loadable modules.', long_desc => 'If a dynamically loadable module needs to be opened and the specified name does not have a directory component (i.e., the name does not contain a slash), the system will search this path for the specified file.', diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c index 6f0ec8757f4..81824151a1f 100644 --- a/src/backend/utils/misc/guc_tables.c +++ b/src/backend/utils/misc/guc_tables.c @@ -33,6 +33,7 @@ #include "access/gin.h" #include "access/slru.h" #include "access/toast_compression.h" +#include "access/toast_dictionary.h" #include "access/twophase.h" #include "access/xlog_internal.h" #include "access/xlogprefetcher.h" diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index c4a61f4d7fb..a2bfcf83f7d 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -807,6 +807,12 @@ #default_table_access_method = 'heap' #default_tablespace = '' # a tablespace name, '' uses the default #default_toast_compression = pglz # pglz, lz4, or zstd +#dictionary_evolution_max_retained = 8 # max TOAST dictionaries kept per + # column by automatic evolution +#dictionary_evolution_min_gain = 0.05 # min fractional size win to adopt a + # retrained TOAST dictionary +#dictionary_evolution_min_interval = 86400 # (s) min time between automatic + # TOAST dictionary retrainings #temp_tablespaces = '' # a list of tablespace names, '' uses # only default tablespace #check_function_bodies = on diff --git a/src/include/access/toast_dictionary.h b/src/include/access/toast_dictionary.h index 41b4fdf5c85..169f801653b 100644 --- a/src/include/access/toast_dictionary.h +++ b/src/include/access/toast_dictionary.h @@ -12,6 +12,7 @@ #ifndef TOAST_DICTIONARY_H #define TOAST_DICTIONARY_H +#include "access/htup.h" #include "utils/relcache.h" #include "varatt.h" @@ -30,16 +31,30 @@ /* * Per-codec dictionary routines. The catalog, lifecycle, training, and the * datum layout above are codec-independent and live in toast_dictionary.c; - * only these three calls differ per codec. compress() lays out the header - * and writes the codec payload after VARHDRSZ_COMPRESSED_DICT; it returns - * NULL if the value did not get smaller. decompress() and - * decompress_slice() read a datum this codec produced. + * only these calls differ per codec. compress() lays out the header and + * writes the codec payload after VARHDRSZ_COMPRESSED_DICT; it returns NULL if + * the value did not get smaller. decompress() and decompress_slice() read a + * datum this codec produced. + * + * train() and compressed_size() exist only for automatic dictionary + * evolution (see toast_dictionary_maybe_evolve()). train() builds a + * dictionary of at most max_dict_bytes from nsamples concatenated samples + * (sizes[] gives each one's length), writing it into dictbuf and returning + * its length, or 0 on failure (a decline, not an error). compressed_size() + * compresses one value against raw dictionary bytes and returns the + * compressed byte count without storing anything, so the evolution policy can + * weigh a candidate dictionary against the current one. Both are codec + * contract points only: the policy never interprets the dictionary bytes. */ typedef struct DictCompressRoutine { varlena *(*compress) (const varlena *value, Oid dictid); varlena *(*decompress) (const varlena *value); varlena *(*decompress_slice) (const varlena *value, int32 slicelength); + size_t (*train) (const char *samples, const size_t *sizes, + int nsamples, size_t max_dict_bytes, void *dictbuf); + size_t (*compressed_size) (const char *dict, size_t dictlen, + const char *val, size_t len); } DictCompressRoutine; /* Current dictionary of rel.attnum, or InvalidOid if none */ @@ -62,4 +77,20 @@ extern varlena *toast_dict_decompress_datum_slice(const varlena *value, extern void RemoveToastDictionaries(Oid relid, AttrNumber attnum); extern void toast_dictionary_retire(Oid relid); +/* + * Automatic dictionary evolution, called from ANALYZE for opted-in columns + * that use a dictionary method. Reuses ANALYZE's already-fetched sample + * rows; trains a candidate, runs a held-out size test, and installs it only + * on a margin win, all in the maintenance process. A no-op for every other + * column. + */ +extern void toast_dictionary_maybe_evolve(Relation onerel, AttrNumber attnum, + HeapTuple *rows, int numrows, + double totalrows); + +/* GUCs controlling automatic dictionary evolution; see toast_dictionary.c */ +extern PGDLLIMPORT double dictionary_evolution_min_gain; +extern PGDLLIMPORT int dictionary_evolution_min_interval; +extern PGDLLIMPORT int dictionary_evolution_max_retained; + #endif /* TOAST_DICTIONARY_H */ diff --git a/src/include/catalog/pg_toast_dictionary.h b/src/include/catalog/pg_toast_dictionary.h index a2abc91ae9c..2d0e164a8a2 100644 --- a/src/include/catalog/pg_toast_dictionary.h +++ b/src/include/catalog/pg_toast_dictionary.h @@ -40,9 +40,13 @@ CATALOG(pg_toast_dictionary,9260,ToastDictionaryRelationId) Oid tdrelid BKI_LOOKUP(pg_class); /* owning relation */ int16 tdattnum; /* owning column */ bool tdcurrent; /* used to compress new values? */ + int64 tdtrainrows; /* reltuples estimate when trained, for + * the auto-evolution growth trigger */ + int64 tdtraintime; /* training time as a TimestampTz, for the + * auto-evolution cooldown */ #ifdef CATALOG_VARLEN /* variable-length fields start here */ - bytea tddict BKI_FORCE_NOT_NULL; /* zstd dictionary content */ + bytea tddict BKI_FORCE_NOT_NULL; /* trained dictionary content */ #endif } FormData_pg_toast_dictionary; diff --git a/src/include/utils/attoptcache.h b/src/include/utils/attoptcache.h index 172055fcd10..e0f7f993633 100644 --- a/src/include/utils/attoptcache.h +++ b/src/include/utils/attoptcache.h @@ -21,6 +21,9 @@ typedef struct AttributeOpts int32 vl_len_; /* varlena header (do not touch directly!) */ float8 n_distinct; float8 n_distinct_inherited; + bool dictionary_evolution; + float8 dictionary_evolution_growth; + int32 dictionary_evolution_min_rows; } AttributeOpts; extern AttributeOpts *get_attribute_options(Oid attrelid, int attnum); diff --git a/src/include/utils/rel.h b/src/include/utils/rel.h index 7b8c2b1e362..f3b322e36ac 100644 --- a/src/include/utils/rel.h +++ b/src/include/utils/rel.h @@ -157,6 +157,18 @@ typedef struct RelationData /* data managed by RelationGetStatExtList: */ List *rd_statlist; /* list of OIDs of extended stats */ + /* + * Single-entry cache of a column's current TOAST compression dictionary + * OID, filled lazily by toast_dictionary_current() to avoid a catalog + * index probe per compressed value. rd_toastdictattnum is the cached + * column (0 means the cache is empty); rd_toastdictoid is its current + * dictionary OID (InvalidOid if the column has none). dict_install() + * issues CacheInvalidateRelcacheByRelid(), which rebuilds the entry and so + * drops this cache, whenever a column's current dictionary changes. + */ + AttrNumber rd_toastdictattnum; + Oid rd_toastdictoid; + /* data managed by RelationGetIndexAttrBitmap: */ bool rd_attrsvalid; /* are bitmaps of attrs valid? */ Bitmapset *rd_keyattr; /* cols that can be ref'd by foreign keys */ diff --git a/src/test/regress/expected/compression_dict_evolve.out b/src/test/regress/expected/compression_dict_evolve.out new file mode 100644 index 00000000000..277f87b2f7c --- /dev/null +++ b/src/test/regress/expected/compression_dict_evolve.out @@ -0,0 +1,152 @@ +-- Tests for automatic TOAST dictionary evolution during ANALYZE: an opted-in +-- column retrains its dictionary from ANALYZE's sample, adopts a candidate +-- only when a held-out size test shows a margin win, rejects churn when no +-- candidate wins, and stops at the retained-dictionary cap. See +-- toast_dictionary_maybe_evolve(). +SELECT NOT(enumvals @> '{zstd}') AS skip_test FROM pg_settings WHERE + name = 'default_toast_compression' \gset +\if :skip_test + \echo '*** skipping TOAST dictionary evolution tests (not supported) ***' + \quit +\endif +CREATE SCHEMA dict_evolve; +SET search_path TO dict_evolve, public; +-- Lenient gates so the test is deterministic: no cooldown and any growth +-- (the row count is held constant below, so the growth gate must accept +-- equality), with a low row floor. The 5% margin keeps its default so the +-- held-out size test is the real adoption decision. +SET dictionary_evolution_min_interval = 0; +-- Two visibly different value distributions. A dictionary trained on one +-- compresses it well and the other poorly, so switching the stored +-- distribution produces a candidate that wins the held-out test. +CREATE FUNCTION docA(i int) RETURNS text LANGUAGE SQL IMMUTABLE AS $$ + SELECT format('{"customer_id": %s, "status": "active", "region": "us-east", ' + '"plan": "enterprise", "features": ["sso", "audit", "backup"], ' + '"env": "production", "tier": "gold", "shard": "a-%s"}', + i, i % 7) +$$; +CREATE FUNCTION docB(i int) RETURNS text LANGUAGE SQL IMMUTABLE AS $$ + SELECT format('warehousepallet' + 'cold-storage%sin-transit' + 'freightwaysstandard', + i, i % 7) +$$; +CREATE TABLE evo (id int, f1 text COMPRESSION zstd_dict) + WITH (autovacuum_enabled = off); +ALTER TABLE evo ALTER COLUMN f1 SET (dictionary_evolution = on, + dictionary_evolution_growth = 1.0, + dictionary_evolution_min_rows = 50); +SELECT 'evo'::regclass::oid AS evo_oid \gset +-- no dictionary yet +SELECT count(*) AS ndicts FROM pg_toast_dictionary WHERE tdrelid = :evo_oid; + ndicts +-------- + 0 +(1 row) + +-- A fixed set of rows, distribution A. ANALYZE adopts the first dictionary +-- unconditionally once the row floor is cleared. +INSERT INTO evo SELECT g, docA(g) FROM generate_series(1, 4000) g; +ANALYZE evo; +SELECT count(*) AS ndicts, count(*) FILTER (WHERE tdcurrent) AS ncurrent + FROM pg_toast_dictionary WHERE tdrelid = :evo_oid; + ndicts | ncurrent +--------+---------- + 1 | 1 +(1 row) + +SELECT oid AS dicta FROM pg_toast_dictionary + WHERE tdrelid = :evo_oid AND tdcurrent \gset +-- Same distribution again: a re-trained candidate does not beat the current +-- dictionary by the margin, so nothing is adopted and the catalog is quiet. +ANALYZE evo; +SELECT count(*) AS ndicts FROM pg_toast_dictionary WHERE tdrelid = :evo_oid; + ndicts +-------- + 1 +(1 row) + +SELECT oid = :dicta AS still_same_current FROM pg_toast_dictionary + WHERE tdrelid = :evo_oid AND tdcurrent; + still_same_current +-------------------- + t +(1 row) + +-- Drift: rewrite every value to distribution B (row count unchanged, so the +-- growth gate still accepts). ANALYZE should find a candidate that beats the +-- A-trained dictionary on the held-out B sample and adopt it: the count rises +-- and the current dictionary changes. +UPDATE evo SET f1 = docB(id); +ANALYZE evo; +SELECT count(*) AS ndicts FROM pg_toast_dictionary WHERE tdrelid = :evo_oid; + ndicts +-------- + 2 +(1 row) + +SELECT oid <> :dicta AS current_changed FROM pg_toast_dictionary + WHERE tdrelid = :evo_oid AND tdcurrent; + current_changed +----------------- + t +(1 row) + +SELECT oid AS dictb FROM pg_toast_dictionary + WHERE tdrelid = :evo_oid AND tdcurrent \gset +-- Stable again on B: ANALYZE must not churn. +ANALYZE evo; +SELECT oid = :dictb AS still_same_current FROM pg_toast_dictionary + WHERE tdrelid = :evo_oid AND tdcurrent; + still_same_current +-------------------- + t +(1 row) + +-- Every value still reads back correctly (each decodes with the dictionary +-- stamped into it, old or new). +SELECT bool_and(f1 = docB(id)) AS all_equal FROM evo; + all_equal +----------- + t +(1 row) + +-- Cap: with a low cap, repeated drift keeps adopting only until the cap is +-- reached, after which evolution declines and the catalog stops growing. The +-- cap message goes to the server log; here we just check the bound holds. +SET dictionary_evolution_max_retained = 3; +DO $$ +DECLARE + i int; +BEGIN + FOR i IN 1..6 LOOP + IF i % 2 = 0 THEN + UPDATE evo SET f1 = docA(id); + ELSE + UPDATE evo SET f1 = docB(id); + END IF; + ANALYZE evo; + END LOOP; +END $$; +SELECT count(*) <= 3 AS within_cap, + count(*) FILTER (WHERE tdcurrent) AS ncurrent + FROM pg_toast_dictionary WHERE tdrelid = :evo_oid; + within_cap | ncurrent +------------+---------- + t | 1 +(1 row) + +SELECT bool_and(f1 = docA(id)) AS all_equal FROM evo; + all_equal +----------- + t +(1 row) + +RESET dictionary_evolution_max_retained; +RESET dictionary_evolution_min_interval; +RESET search_path; +DROP SCHEMA dict_evolve CASCADE; +NOTICE: drop cascades to 3 other objects +DETAIL: drop cascades to function dict_evolve.doca(integer) +drop cascades to function dict_evolve.docb(integer) +drop cascades to table dict_evolve.evo diff --git a/src/test/regress/expected/compression_dict_evolve_1.out b/src/test/regress/expected/compression_dict_evolve_1.out new file mode 100644 index 00000000000..e35cd728759 --- /dev/null +++ b/src/test/regress/expected/compression_dict_evolve_1.out @@ -0,0 +1,11 @@ +-- Tests for automatic TOAST dictionary evolution during ANALYZE: an opted-in +-- column retrains its dictionary from ANALYZE's sample, adopts a candidate +-- only when a held-out size test shows a margin win, rejects churn when no +-- candidate wins, and stops at the retained-dictionary cap. See +-- toast_dictionary_maybe_evolve(). +SELECT NOT(enumvals @> '{zstd}') AS skip_test FROM pg_settings WHERE + name = 'default_toast_compression' \gset +\if :skip_test + \echo '*** skipping TOAST dictionary evolution tests (not supported) ***' +*** skipping TOAST dictionary evolution tests (not supported) *** + \quit diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index 0f1e01ad506..23417216cec 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -128,7 +128,7 @@ test: partition_join partition_prune reloptions hash_part indexing partition_agg # ---------- # Another group of parallel tests (compression) # ---------- -test: compression compression_lz4 compression_pglz compression_zstd compression_zstd_dict compression_lz4_dict cluster +test: compression compression_lz4 compression_pglz compression_zstd compression_zstd_dict compression_lz4_dict compression_dict_evolve cluster # event_trigger depends on create_am and cannot run concurrently with # any test that runs DDL diff --git a/src/test/regress/sql/compression_dict_evolve.sql b/src/test/regress/sql/compression_dict_evolve.sql new file mode 100644 index 00000000000..ed85aa94640 --- /dev/null +++ b/src/test/regress/sql/compression_dict_evolve.sql @@ -0,0 +1,111 @@ +-- Tests for automatic TOAST dictionary evolution during ANALYZE: an opted-in +-- column retrains its dictionary from ANALYZE's sample, adopts a candidate +-- only when a held-out size test shows a margin win, rejects churn when no +-- candidate wins, and stops at the retained-dictionary cap. See +-- toast_dictionary_maybe_evolve(). + +SELECT NOT(enumvals @> '{zstd}') AS skip_test FROM pg_settings WHERE + name = 'default_toast_compression' \gset +\if :skip_test + \echo '*** skipping TOAST dictionary evolution tests (not supported) ***' + \quit +\endif + +CREATE SCHEMA dict_evolve; +SET search_path TO dict_evolve, public; + +-- Lenient gates so the test is deterministic: no cooldown and any growth +-- (the row count is held constant below, so the growth gate must accept +-- equality), with a low row floor. The 5% margin keeps its default so the +-- held-out size test is the real adoption decision. +SET dictionary_evolution_min_interval = 0; + +-- Two visibly different value distributions. A dictionary trained on one +-- compresses it well and the other poorly, so switching the stored +-- distribution produces a candidate that wins the held-out test. +CREATE FUNCTION docA(i int) RETURNS text LANGUAGE SQL IMMUTABLE AS $$ + SELECT format('{"customer_id": %s, "status": "active", "region": "us-east", ' + '"plan": "enterprise", "features": ["sso", "audit", "backup"], ' + '"env": "production", "tier": "gold", "shard": "a-%s"}', + i, i % 7) +$$; +CREATE FUNCTION docB(i int) RETURNS text LANGUAGE SQL IMMUTABLE AS $$ + SELECT format('warehousepallet' + 'cold-storage%sin-transit' + 'freightwaysstandard', + i, i % 7) +$$; + +CREATE TABLE evo (id int, f1 text COMPRESSION zstd_dict) + WITH (autovacuum_enabled = off); +ALTER TABLE evo ALTER COLUMN f1 SET (dictionary_evolution = on, + dictionary_evolution_growth = 1.0, + dictionary_evolution_min_rows = 50); +SELECT 'evo'::regclass::oid AS evo_oid \gset + +-- no dictionary yet +SELECT count(*) AS ndicts FROM pg_toast_dictionary WHERE tdrelid = :evo_oid; + +-- A fixed set of rows, distribution A. ANALYZE adopts the first dictionary +-- unconditionally once the row floor is cleared. +INSERT INTO evo SELECT g, docA(g) FROM generate_series(1, 4000) g; +ANALYZE evo; +SELECT count(*) AS ndicts, count(*) FILTER (WHERE tdcurrent) AS ncurrent + FROM pg_toast_dictionary WHERE tdrelid = :evo_oid; +SELECT oid AS dicta FROM pg_toast_dictionary + WHERE tdrelid = :evo_oid AND tdcurrent \gset + +-- Same distribution again: a re-trained candidate does not beat the current +-- dictionary by the margin, so nothing is adopted and the catalog is quiet. +ANALYZE evo; +SELECT count(*) AS ndicts FROM pg_toast_dictionary WHERE tdrelid = :evo_oid; +SELECT oid = :dicta AS still_same_current FROM pg_toast_dictionary + WHERE tdrelid = :evo_oid AND tdcurrent; + +-- Drift: rewrite every value to distribution B (row count unchanged, so the +-- growth gate still accepts). ANALYZE should find a candidate that beats the +-- A-trained dictionary on the held-out B sample and adopt it: the count rises +-- and the current dictionary changes. +UPDATE evo SET f1 = docB(id); +ANALYZE evo; +SELECT count(*) AS ndicts FROM pg_toast_dictionary WHERE tdrelid = :evo_oid; +SELECT oid <> :dicta AS current_changed FROM pg_toast_dictionary + WHERE tdrelid = :evo_oid AND tdcurrent; +SELECT oid AS dictb FROM pg_toast_dictionary + WHERE tdrelid = :evo_oid AND tdcurrent \gset + +-- Stable again on B: ANALYZE must not churn. +ANALYZE evo; +SELECT oid = :dictb AS still_same_current FROM pg_toast_dictionary + WHERE tdrelid = :evo_oid AND tdcurrent; + +-- Every value still reads back correctly (each decodes with the dictionary +-- stamped into it, old or new). +SELECT bool_and(f1 = docB(id)) AS all_equal FROM evo; + +-- Cap: with a low cap, repeated drift keeps adopting only until the cap is +-- reached, after which evolution declines and the catalog stops growing. The +-- cap message goes to the server log; here we just check the bound holds. +SET dictionary_evolution_max_retained = 3; +DO $$ +DECLARE + i int; +BEGIN + FOR i IN 1..6 LOOP + IF i % 2 = 0 THEN + UPDATE evo SET f1 = docA(id); + ELSE + UPDATE evo SET f1 = docB(id); + END IF; + ANALYZE evo; + END LOOP; +END $$; +SELECT count(*) <= 3 AS within_cap, + count(*) FILTER (WHERE tdcurrent) AS ncurrent + FROM pg_toast_dictionary WHERE tdrelid = :evo_oid; +SELECT bool_and(f1 = docA(id)) AS all_equal FROM evo; + +RESET dictionary_evolution_max_retained; +RESET dictionary_evolution_min_interval; +RESET search_path; +DROP SCHEMA dict_evolve CASCADE; -- 2.50.1