Re: ZSTD TOAST compression, and an extensible compression method encoding

Hi Manu,

The patches I attached as v8 came from the wrong branch, I was rushing
and I shouldn't have been. They were v7 plus the earlier 0004, so they
had neither your context reuse, nor the comments, nor your name. Sorry
for the noise. v9, attached, is what my last email described. Thanks for
the repro. I ran repro-dctx-reset.sh unchanged against v9 and against v9
with the ZSTD_DCtx_reset() call removed:

- v9: all checks true, B2 included, the three expected "compressed zstd
data is corrupt" errors and no others: "RESULT pass".
- Without the reset: B2 ("slice of another value after an early stop")
comes back false with no error, while the full reads A2, A3 and B3
stay true. The slices that should have succeeded after an error fail
instead, which is the false error I saw before.

So both cases reproduce here, and the reset fixes both.

I've added Dharin, Peter and Robert, because this series overlaps with
Dharin's "Add zstd compression for TOAST using extended header format"
thread [1] from December, which nobody here had referred to yet. That
thread is where several pieces of this one came from:

- Dharin proposed a decoded in-memory form of an external TOAST pointer
[2]. Michael took that direction for the Oid8 work [3], and it is now
toast_external_info_get() and toast_external_data on master
(6e58d6356fc). 0001 extends it with compress_method, so v9-0001 now
carries Suggested-by: Dharin Shah (or maybe just Co-authored-by?)

- Michael's preference there was one new vartag_external per method,
keeping the TOAST pointer at its plain size [4], and Dharin's second
version did that with an external-only VARTAG_ONDISK_ZSTD [5]. v7
went the other way: one pair of long-form tags plus a method byte.
The 0002 commit message now says why. An inline compressed datum has
no vartag, so a tag per method can't give inline values a third
method. And vartag_external is a uint8 shared with the in-memory
pointer kinds, so it shouldn't take one tag per method. The cost is
one byte, in the pointer or the inline header, for methods other than
pglz and lz4. Michael said upthread that this cost doesn't bother
him [6]; I'm raising it here so the earlier preference isn't
overlooked.

- Peter asked for numbers [7]. For compression ratio, Dharin measured
170MB of mixed data at 22MB with zstd against 36MB with pglz or lz4
[5], and Nikhil's tweet data set in the first thread gave 210MB with
zstd against 259MB pglz and 297MB lz4, or 114MB with a dictionary
[8]. For speed, your and my numbers on context reuse are earlier in
this thread. None of these is an end-to-end benchmark, and the
pgbench run you mentioned would be the first. If I have time I'll
give it a try.

v9 is rebased on current master (0f3b42ce806):

0001 Decode the compression method in toast_external_info_get()
0002 Allow more than four TOAST compression methods
0003 Add zstd as a TOAST compression method (with context reuse)
0004 Add zstd compression dictionaries with a column-local lifecycle

Training is an explicit call for now; doing it from ANALYZE's sample is
the obvious next step once the lifecycle is agreed on. Well, that and
doing some benchmarking.

best.

-greg

[1] https://postgr.es/m/CAOj6k6dy2CRVA6Lsb5N59zE-7KNVKt=oYwWyg8ULK8zOOY8e7A@mail.gmail.com
[2] https://postgr.es/m/CAOj6k6dEVi0NvLjMLDhyrJS_n_NZO5D_OU89AO1u53u6NCDDwQ@mail.gmail.com
[3] https://postgr.es/m/aW8keI6WZxywRQIt@paquier.xyz
[4] https://postgr.es/m/aUOgmypuun0rc2xO@paquier.xyz
[5] https://postgr.es/m/CAOj6k6f2B3hNxDcnB5AgHX4kaTW8XTAfMAjRx4upDBOugxqF4w@mail.gmail.com
[6] https://postgr.es/m/arS_NVVZZ2XsfCYU@paquier.xyz
[7] https://postgr.es/m/4c70b0f6-5aba-464c-b145-464a620c1222@eisentraut.org
[8] https://postgr.es/m/CAFAfj_GACKVftwuRjy3Ls-1Xc3ojUUbVh=Rm7KpRuYbaS=uLPg@mail.gmail.com