Re: ZSTD TOAST compression, and an extensible compression method encoding

Hi Nikhil,

> v4 attached, rebased on e27f3b2cad7 and renumbered now that the first
> patch is in:

I ran some checks and numbers on v4, on master 3c5d9d914fa. Scripts
and the full tables are attached.

On-disk format. The cover letter says pglz and lz4 values keep their
representation bit for bit. Writing the same data with master and
with v4, the raw bytes of the inline compressed datums (pageinspect)
and of the TOAST chunks hash the same, for pglz and lz4. pg_upgrade
from master to v4 keeps every value, and verify_heapam(check_toast)
finds nothing. It is also clean on v4 for a column holding pglz, lz4
and zstd values at once, and after VACUUM FULL, UPDATE and DELETE on
a zstd table.

No slowdown for the existing methods. v4 against master with lz4,
paired within each of 6 rounds: the median ratio is between 0.97 and
1.06 for every point, and no point is slower in all rounds. The two
rounds with pglz look the same.

zstd itself, on -O2 builds, hot cache, 200 MB per case, sizes as a
percentage of the raw data (pglz / lz4 / zstd):

- SGML docs in 32 kB values: 35.7% / 41.0% / 26.9%
- JSON documents in 1 MB values: 12.1% / 18.0% / 4.6%
- base64 text in 32 kB values: 102.9% / 102.9% / 77.0%

The last one is worth noting: pglz and lz4 get nothing out of base64
or hex text, and zstd saves about a quarter.

The cost is on reads of small values. For 50,000 JSON values of
4 kB, reading them all takes 140 ms with zstd against 53 ms with lz4,
and a 100-byte prefix costs the same as the whole value (140 ms,
against 23 ms with lz4).

One thing in 0004 that is easy to improve:
zstd_decompress_datum_slice() creates and frees a ZSTD_DCtx for every
value. The attached diff keeps one per backend, reset on each use,
and uses it in both decompression paths. zstd, 4 rounds, ratio of
the diff to v4:

- 32 kB values, slice and prefix: 0.66 to 0.77, faster in every round
- 32 kB values, full read: 0.87 to 0.89 for docs and base64
- 1 MB base64 values, slice: 0.70
- 4 kB values: 0.94 to 0.97

make check passes with it. At 4 kB the prefix still costs as much as
the whole value, so what is left there is per frame, not the context.

Regards,
Manu