Re: ZSTD TOAST compression, and an extensible compression method encoding

Hi Greg,

On Sat, Oct 24, 2026 at 6:33 PM Greg Burd <greg(at)burd(dot)me> wrote:
> If you want to compare setups, send your scripts and per-run numbers
> and I'll diff them against mine.

Thanks for re-measuring on bare metal, and for the new 0003 sentence.
Scripts and numbers are attached; first, the 1MB slice.

It is the database encoding. bench.sh runs on a SQL_ASCII database on
purpose. With a multibyte encoding, text_substring() asks detoast for
bytes [0, (start + len - 1) * eml) (varlena.c:666-718), and a request
at least as long as the raw size becomes a full decompression
(detoast.c:529); lines as of master 42e96cf2fe0, and v10 changes
neither. I checked it in the server, with breakpoints at the entry of
zstd_decompress_datum() and zstd_decompress_datum_slice(), on 20
values of 1 MB:

- UTF8: substr(v, 500000, 100) enters zstd_decompress_datum() 20
times and the slice function never.
- SQL_ASCII: the same query enters zstd_decompress_datum_slice() 20
times, with slicelength 500099.
- substr(v, 1, 100) is a slice in both (slicelength 400 and 100).

By the same code, substr(v, 20000, 100) on UTF8 asks for 80,396
bytes, more than a 32 or 46 kB value, so it is a full decompression
too; I did not run that one.

So, if your database is UTF8, your 1MB slice(at)500k (1.001) is a full
read, like our 1 MB full read (1.000 [0.985, 1.006]); your 32kB
slice(at)20k (0.953) sits with your full read (0.956); and your 32kB
prefix (0.614) is the one that stays a slice.

The 172 us per value that the reuse saves on our 1 MB slice (from the
median times) track page faults. On SQL_ASCII the backend takes 416
page faults per value on that slice with a fresh context and none with
the reused one, the same in three repetitions, about 0.4 us per fault.
The 1 MB prefix (0 and 0 faults) and full read (631 and 631) show no
fault difference, and their timings barely move (0.980 and 1.000).
These counts come from the builds I still have: v7 plus the reuse,
whose three zstd decode functions are line for line those of v10-0003,
and v7, whose two match v10 with the control diff below.

The 32 kB rows are not the same test: if I read your setup right,
they run on the 80k x 46kB docs, and ours on 32,000-byte docs, json
and random values. Size and content both differ, so I would not
read 0.614 vs our 0.70-0.76 as a disagreement.

Our setup: Ryzen AI 9 HX 370 laptop, 8 CPUs pinned, performance
profile; -O2 without asserts, gcc 16.2.1, zstd 1.5.7, --with-lz4
--with-zstd; shared_buffers 2GB; hot cache; SQL_ASCII, locale C.

Attached:
- bench.sh.txt: the benchmark as used, with only local paths replaced.
- run_v9_cite.sh.txt: the rounds, both builds back to back, order
alternating. Our lab's idle-CPU check is stubbed out; it only
flagged rounds, all 38 were clean, so no number depends on it.
- cite_v9.py.txt, cite_v9_runs.py.txt: the summaries (rename to .py;
the second imports the first).
- per-run-v9.txt: per round and point, the median of 5 reps for each
build and the ratio, runs A, B and C.
- nocfbot-v9n-without-reuse.diff.txt: our v7 control on top of 0003.
It also applies on top of v10-0004.
- branch_faults.sh.txt and branch_faults-out.txt: the breakpoint
and page-fault counts above, and their output.

Regards,
Manu