ZSTD TOAST compression, v14: audit of the containment checks ============================================================ Tree: master fa5ecff759b + v14-0001..0005. File:line below are in that tree. The table was written from reading the code, before any of the tests in the last section were run. What is audited. 0004 keeps a dictionary datum inside its column with two mechanisms: fill_val() sets HEAP_HASEXTERNAL for a tuple that holds one (heaptuple.c:372-374), so everything that already flattens a tuple with TOAST pointers also reaches it, and toast_tuple_init() decompresses it when it is stored anywhere but its own column (toast_helper.c:205-221). The toaster does not look inside composite, array or range values, so a dictionary datum can escape where code decides "is this value self-contained?" by testing ONE datum with VARATT_IS_EXTERNAL() instead of the tuple flag, and then keeps or copies it. Hannu's check_domain_for_new_field() finding is of that kind. Scope: every line in src/ and contrib/ (without src/test) that uses VARATT_IS_EXTERNAL[_ONDISK|_INDIRECT|_EXPANDED|_NON_EXPANDED](), VARATT_IS_EXTENDED(), VARATT_IS_COMPRESSED() or HeapTuple[Header]HasExternal(): 141 lines in 47 files, grouped below by condition or function, plus 24 places that decide self-containment without those macros. OK = handles dictionary datums (how). NA = not a containment decision (why). MISSING = takes the wrong branch for a dictionary datum. A. Composite values and expanded records MISSING expandedrecord.c:1553 check_domain_for_new_field: the new value is tested with VARATT_IS_EXTERNAL() only, so dummy_erh is flattened for the domain check without inlining it (Hannu, 02-11) OK expandedrecord.c:472, 523 expanded_record_set_tuple: tuple flag OK expandedrecord.c:707 ER_get_flat_size: VARATT_IS_COMPRESSED_DICT OK expandedrecord.c:1160, 1191 expanded_record_set_field_internal: same OK expandedrecord.c:1307, 1320 expanded_record_set_fields: same OK expandedrecord.c:1523 check_domain_for_new_field, old fields: copies erh's flag OK expandedrecord.c:1617 check_domain_for_new_tuple: tuple flag OK execTuples.c:2517 HeapTupleHeaderGetDatum: flag -> flatten OK heaptuple.c:1009 heap_copy_tuple_as_datum: flag -> flatten OK heaptuple.c:328-330 fill_val: sets HEAP_HASEXTERNAL (:372-374) OK heaptoast.c:378 toast_flatten_tuple: VARATT_IS_COMPRESSED_DICT OK heaptoast.c:493-494 toast_flatten_tuple_to_datum: external or compressed -> detoast_attr OK heaptoast.c:594 toast_build_flattened_tuple: as :378 NA expandedrecord.c:634 Assert (composite datums are flat) NA execExprInterp.c:3757 FieldSelect on an expanded record (a read) NA heaptuple.c:249 heap_compute_data_size: expanded only NA pl_exec.c:836, 1092, 7698 expanded-object checks B. Values that outlive the transaction or the lock OK pl_exec.c:8920 assign_simple_var (non-atomic) OK tstoreReceiver.c:163 WITH HOLD cursors OK datum.c:201 datumTransfer / SPI_datumTransfer (v13) OK detoast.c:51-59, 67-91, 114 detoast_external_attr: on disk, indirect (v13) and inline dictionary datums OK catcache.c:2198 tuple flag (and catalogs cannot use these methods) NA detoast.c:101 expanded object, flattened NA datum.c:144 datumCopy: verbatim, as a TOAST pointer; in-transaction copies NA datum.c:474, 523 datumSerialize: workers of the same query NA execTuples.c:199, 235 slot materialization: expanded only NA expandeddatum.c:34; arrayfuncs.c:1852, 2273; array_expanded.c:76, 410 expanded-form dispatch (elements: section E) C. Storing into a table, an index, or another table's TOAST OK toast_helper.c:143 toast_tuple_init: a foreign external value is fetched with detoast_external_attr, which decompresses it OK toast_helper.c:168-174 toast_dictionary_keep(); the UPDATE fast path needs an old tuple of the same relation, no relrewrite, and byte-identical datums OK heapam.c:2263-2266, 3848-3854 insert / update: tuple flag -> toaster OK heapam.c:9381, 9434 ExtractReplicaIdentity: flag -> flatten OK rewriteheap.c:616-619 raw_heap_insert: flag -> toaster (relrewrite) OK indextuple.c:108 index_form_tuple: decompresses (0004) OK indextuple.c:121 recompression without a rel: plain zstd (toast_internals.c:89-93), although index.c:360 copies attcompression OK brin_tuple.c:209, 221 as indextuple.c OK repack.c:3101; pgrepack.c:214, 256-275 REPACK (CONCURRENTLY): the dictionary datum travels inside the tuple and reaches toast_tuple_init with relrewrite set NA toast_helper.c:79-82, 249-252, 383 unchanged external value of the same relation; size heuristics; deletion NA heapam.c:3183-3185, 4639, 6526 deletion; has_external for replica identity (a dictionary datum is inline, so in WAL already); speculative abort NA toast_internals.c:332 toast_save_datum: keep() ran before; header validation is Hannu's point 2 (matrix, 2b) NA toast_internals.c:53, 103-104, 241, 295, 569; heaptoast.c:720; repack.c:3047 Asserts, deletion, slice fetch, indirect D. Reading, reporting, decoding NA detoast.c:149-199, 257-325, 388, 445, 519, 560, 617-640, 678-701 detoast_attr[_slice], sizes, preconditions NA fmgr.c:1800, 1809, 1832 pg_detoast_datum*: decompress everything NA varlena.c:716-717, 4292 text_substring; a reporting function NA toast_compression.c:525, 534 toast_get_compression_id (reporting) NA reorderbuffer.c:5187, 5306, 5345; proto.c:815; pgoutput.c:1405-1406; test_decoding.c:594 decoding: the datum is inline in WAL, and plugins decompress under the historic snapshot, which still sees a retired dictionary while the slot holds catalog_xmin NA inv_api.c:142; ts_selfuncs.c:238 pg_largeobject is a catalog; Assert NA verify_heapam.c:1621, 1751-1755, 1794; verify_nbtree.c:2873-2892; pageinspect heapfuncs.c:371-373 checks; index tuples never hold one NA toast_compression.h:93, 123; detoast.h:28, 69; htup_details.h:576, 753; pl_exec.c:8726; heaptoast.c:346 definitions, comments E. Places that decide without the macros OK arrayfuncs.c:327, 1526, 2271, 3296, 3538, 6216, 6471-6473 array_in, array_recv, array_set_element, array_map, construct_md_array, array_fill, array_replace: PG_DETOAST_DATUM OK arrayfuncs.c:5400 accumArrayResult: PG_DETOAST_DATUM_COPY OK arrayfuncs.c:2501 array_set_element_expanded: only caller is :2276, after :2271 OK rangetypes.c:2033, 2043 range_serialize: PG_DETOAST_DATUM_PACKED, so the :2960 guard (toast pointers only) is never reached by a dictionary datum; multiranges store serialized ranges OK spgdoinsert.c:1959, 1973 SP-GiST leaf and INCLUDE values OK analyze.c:2243, 2590; mcv.c:764; extended_stats.c:1148 statistics OK makefuncs.c:366; clauses.c:5995 makeConst, evaluate_expr OK spi.c:1097; execTuples.c:2017 SPI_returntuple, ExecFetchSlotHeapTupleDatum: tuple flag NA tuplestore.c, tuplesort.c verbatim within the query; WITH HOLD goes through tstoreReceiver.c:163 NA postgres_fdw.c:4591, 4622, 5198; dblink; copyto.c:526; logical replication values sent through output/send functions Result from reading: 1 MISSING (expandedrecord.c:1553), none other. Tests (run after the table was written) --------------------------------------- One inline zstd_dict value (326 B) taken through each path while its dictionary is current; then (A) a second dictionary trained and VACUUM FULL on the source, which deletes the first, and (B) DROP TABLE on the source; every copy read back against the value's md5. v14 = cassert+ASan; v14+F = the same with Hannu's two suggested changes. "stored" is how the copied value, or the field / element inside it, is stored. path stored v14 after A, B v14+F ROW(d, 1)::comp_t plain ok, ok ok, ok whole-row Var plain ok, - (1) ok, - (1) UPDATE ... SET v.a = d plain ok, ok ok, ok SQL function returning a row plain ok, ok ok, ok PL/pgSQL RETURN record plain ok, ok ok, ok record field := d plain ok, ok ok, ok SELECT * INTO record plain ok, - (1) ok, - (1) domain over composite, field := d, CHECK stores VALUE (Hannu's case) zstd_dict ERROR (2), ERROR plain; ok, ok same, SELECT ... INTO plain ok, ok ok, ok same, whole ROW() assigned plain ok, ok ok, ok domain over text, CHECK stores VALUE lz4 ok, ok ok, ok ARRAY[d], array_agg, a[1] := d, array_append, array_fill, array_replace plain ok, ok (each) ok, ok (each) range and multirange over text plain ok, ok ok, ok ADD COLUMN ... DEFAULT f() (attmissingval) plain ok, ok ok, ok COPY BINARY out and in lz4 ok, ok ok, ok postgres_fdw INSERT (loopback) lz4 ok, ok ok, ok dblink_exec INSERT lz4 ok, ok ok, ok btree index on the column, index-only scan zstd (3) - - (1) the column has the source's row type, so DROP TABLE ... CASCADE drops it. (2) "compression dictionary N does not exist". (3) index_form_tuple() decompresses it and recompresses without a rel. No failed assertion and no ASan report in either build. The rest of the paths in the table are covered by the containment matrix (holdable cursors, procedure variables, SPI_datumTransfer, indirect pointers, logical decoding, statistics, partitions, REPACK (CONCURRENTLY)).