Re: Support for 8-byte TOAST values (aka the TOAST infinite loop problem)

Hi Michael,

I'm reviewing your patches(toast_64bit_v2 branch) and have prepared
two incremental patches that add ZSTD compression support. While doing
this I made a small refactoring in heaptoast.c (around
ToastTupleContext) to make adding additional TOAST pointer formats
easier.

Added two new on-disk external vartags to support new compression
methods: one using an Oid value identifier and one using a 64‑bit
(int8) identifier for toast table types. This lets us support extended
compression methods for both existing Oid‑based TOAST tables and a
variant that needs a wider ID space.

typedef enum vartag_external
{
VARTAG_INDIRECT = 1,
VARTAG_EXPANDED_RO = 2,
VARTAG_EXPANDED_RW = 3,
VARTAG_ONDISK_INT8 = 4,
VARTAG_ONDISK_CE_OID = 5,
VARTAG_ONDISK_CE_INT8 = 6,
VARTAG_ONDISK_OID = 18
} vartag_external;

Two new ondisk TOAST pointer structs carrying an va_ecinfo field for
extended compression methods:

typedef struct varatt_external_ce_oid
{
int32 va_rawsize; /* Original data size (includes header) */
uint32 va_extinfo; /* External saved size (without header) and
* VARATT_CE_FLAG in top 2 bits */
uint32 va_ecinfo; /* Extended compression info */
Oid va_valueid; /* Unique ID of value within TOAST table */
Oid va_toastrelid; /* RelID of TOAST table containing it */
} varatt_external_ce_oid;

typedef struct varatt_external_ce_int8
{
int32 va_rawsize; /* Original data size (includes header) */
uint32 va_extinfo; /* External saved size (without header) and
* VARATT_CE_FLAG in top 2 bits */
uint32 va_ecinfo; /* Extended compression info */

/*
* Unique ID of value within TOAST table, as two uint32 for alignment and
* padding.
*/
uint32 va_valueid_lo;
uint32 va_valueid_hi;
Oid va_toastrelid; /* RelID of TOAST table containing it */
} varatt_external_ce_int8;

The inline (varattrib_4b) format extension (discussed in [1]) is
included; I made one adjustment: the compression id field is now a
4‑byte va_ecinfo field (instead of 1 byte) for structural consistency
with the extended TOAST pointer formats.

typedef union
{
struct /* Normal varlena (4-byte length) */
{
uint32 va_header;
char va_data[FLEXIBLE_ARRAY_MEMBER];
} va_4byte;
struct /* Compressed-in-line format */
{
uint32 va_header;
uint32 va_tcinfo; /* Original data size (excludes header) and
* compression method; see va_extinfo */
char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
} va_compressed;
struct
{
uint32 va_header;
uint32 va_tcinfo; /* Original data size (excludes header) and
* compression method or VARATT_CE_FLAG flag;
* see va_extinfo */
uint32 va_ecinfo; /* Extended compression info: 32-bit field
* where only the lower 8 bits are used for
* compression method. Upper 24 bits are
* reserved/unused. Lower 8 bits layout: Bits
* 7–1: encode (cmid − 2), so cmid is
* [2…129] Bit 0: flag for extra metadata
*/
char va_data[FLEXIBLE_ARRAY_MEMBER];
} va_compressed_ext;
} varattrib_4b;

[1] https://www.postgresql.org/message-id/flat/CAFAfj_HX84EK4hyRYw50AOHOcdVi-%2BFFwAAPo7JHx4aShCvunQ%40mail.gmail.com

Please review it and let me know if you have any questions or
feedback? Thank you!

v26-0014-Design-to-extend-the-varattrib_4b-and-toast-poin.patch:
Design proposal covering varattrib_4b, TOAST pointer layouts, and
related macro updates.
v26-0015-Implement-Zstd-compression-no-dictionary-support.patch: Plain
ZSTD (non dict) support and few basic tests.

--
Nikhil Veldanda