Re: ZStandard (with dictionaries) compression support for TOAST compression

Hi Robert

> But I don't quite understand the point of this
> response: it seems like you're just restating what the design does
> without really justifying it. The question here isn't whether a 3-byte
> header can describe a length up to 16MB; I think we all know our
> powers of two well enough to agree on the answer to that question. The
> question is whether it's a good use of 3 bytes, and I don't think it
> is.

My initial decision to include a 3‑byte length field was driven by two goals:
1. Avoid introducing separate callbacks for each algorithm.
2. Provide a single, algorithm-agnostic mechanism for handling
metadata length.

After re-evaluating based on your feedback, I agree that the fixed
overhead of a 3-byte length field outweighs its benefit; per-algorithm
callbacks deliver the same functionality while saving three bytes per
datum.

> I did consider the fact that future compression algorithms might want
> to use variable-length headers; but I couldn't see a reason why we
> shouldn't let each of those compression algorithms decide for
> themselves how to encode whatever information they need. If a
> compression algorithm needs a variable-length header, then it just
> needs to make that header self-describing. Worst case scenario, it can
> make the first byte of that variable-length header a length byte, and
> then go from there; but it's probably possible to be even smarter and
> use less than a full byte. Say for example we store a dictionary ID
> that in concept is a 32-bit quantity but we use a variable-length
> integer representation for it. It's easy to see that we shouldn't ever
> need more than 3 bits for that so a full length byte is overkill and,
> in fact, would undermine the value of a variable-length representation
> rather severely. (I suspect it's a bad idea anyway, but it's a worse
> idea if you burn a full byte on a length header.)
>

I agree. Each compression algorithm can decide its own metadata size
overhead. Callbacks can provide this information as well rather than
storing in fixed length bytes(3 bytes). The revised patch introduces a
"toast_cmpid_meta_size(const varatt_cmp_extended *hdr)", which
calculates the metadata size.

> But there's an even larger question here too, which is why we're
> having some kind of discussion about generalized metadata when the
> current project seemingly only requires a 4-byte dictionary OID. If
> you have some other use of this space in mind, I don't think you've
> told us what it is. If you don't, then I'm not sure why we're
> designing around an up-to-16MB variable-length quantity when what we
> have before us is a 4-byte fixed-length quantity.

This project only requires 4 bytes of fixed-size metadata to store the
dictionary ID.

Updated design for extending varattrib_4b compression

1. extensible header

/*
* varatt_cmp_extended: an optional per‐datum header for extended
compression method.
* Only used when va_tcinfo's top two bits are "11".
*/
typedef struct varatt_cmp_extended
{
uint8 cmp_alg;
char cmp_meta[FLEXIBLE_ARRAY_MEMBER]; /*
algorithm‐specific metadata */
} varatt_cmp_extended;

2. Algorithm registry and metadata size dispatch

static inline uint32
unsupported_meta_size(const varatt_cmp_extended *hdr)
{
elog(ERROR, "toast_cmpid_meta_size called for unsupported
compression algorithm");
return 0; /* unreachable */
}

/* no metadata for plain-ZSTD */
static inline uint32
zstd_nodict_meta_size(const varatt_cmp_extended *hdr)
{
return 0;
}

static inline uint32
zstd_dict_meta_size(const varatt_cmp_extended *hdr)
{
return sizeof(Oid);
}

/*
* TOAST compression methods enumeration.
*
* NAME : algorithm identifier
* VALUE : enum value
* META-SIZE-FN : Calculates algorithm metadata size.
*/
#define TOAST_COMPRESSION_LIST \
X(PGLZ, 0, unsupported_meta_size) \
X(LZ4, 1, unsupported_meta_size) \
X(ZSTD_NODICT, 2, zstd_nodict_meta_size) \
X(ZSTD_DICT, 3, zstd_dict_meta_size) \
X(INVALID, 4, unsupported_meta_size) /* sentinel */

/* Compression algorithm identifiers */
typedef enum ToastCompressionId
{
#define X(name,val,fn) TOAST_##name##_COMPRESSION_ID = (val),
TOAST_COMPRESSION_LIST
#undef X
} ToastCompressionId;

/* lookup table to check if compression method uses extended format */
static const bool toast_cmpid_extended[] = {
#define X(name,val,fn) \
/* PGLZ, LZ4 don't use extended format */ \
[TOAST_##name##_COMPRESSION_ID] = \
((val) != TOAST_PGLZ_COMPRESSION_ID && \
(val) != TOAST_LZ4_COMPRESSION_ID && \
(val) != TOAST_INVALID_COMPRESSION_ID),
TOAST_COMPRESSION_LIST
#undef X
};

#define TOAST_CMPID_EXTENDED(alg) (toast_cmpid_extended[alg])

/*
* Prototype for a per-datum metadata-size callback:
* given a pointer to the extended header, return
* how many metadata bytes follow it.
*/
typedef uint32 (*ToastMetaSizeFn) (const varatt_cmp_extended *hdr);

/* Callback table—indexed by ToastCompressionId */
static const ToastMetaSizeFn toast_meta_size_fns[] = {
#define X(name,val,fn) [TOAST_##name##_COMPRESSION_ID] = fn,
TOAST_COMPRESSION_LIST
#undef X
};

/* Calculates algorithm metadata size */
static inline uint32
toast_cmpid_meta_size(const varatt_cmp_extended *hdr)
{
Assert(hdr != NULL);
return toast_meta_size_fns[hdr->cmp_alg] (hdr);
}

Each compression algorithm provides a static callback that returns the
size of its metadata, given a pointer to the varatt_cmp_extended
header. Algorithms with fixed-size metadata return a constant, while
algorithms with variable-length metadata are responsible for defining
and parsing their own internal headers to compute the metadata size.

3. Resulting on-disk layouts for zstd

ZSTD (nodict) — datum on‑disk layout

+----------------------------------+
| va_header (uint32) |
+----------------------------------+
| va_tcinfo (uint32) | ← top two bits = 11 (extended)
+----------------------------------+
| cmp_alg (uint8) | ← (ZSTD_NODICT)
+----------------------------------+
| compressed bytes … | ← ZSTD frame
+----------------------------------+

ZSTD(dict) — datum on‑disk layout

+----------------------------------+
| va_header (uint32) |
+----------------------------------+
| va_tcinfo (uint32) | ← top two bits = 11 (extended)
+----------------------------------+
| cmp_alg (uint8) | ← (ZSTD_DICT)
+----------------------------------+
| dict_id (uint32) | ← dictionary OID
+----------------------------------+
| compressed bytes … | ← ZSTD frame
+----------------------------------+

I hope this updated design addresses your concerns. I would appreciate
any further feedback you may have. Thanks again for your guidance—it's
been very helpful.

v20-0001-varattrib_4b-design-proposal-to-make-it-extended.patch:
varattrib_4b extensibility – adds varatt_cmp_extended, metadata size
dispatch and useful macros; behaviour unchanged.
v20-0002-zstd-nodict-compression.patch: Plain ZSTD (non dict) support.

--
Nikhil Veldanda