[PATCH] Add zstd compression for TOAST using extended header format

From: Dharin Shah <dharinshah95(at)gmail(dot)com>
To: pgsql-hackers(at)postgresql(dot)org
Subject: [PATCH] Add zstd compression for TOAST using extended header format
Date: 2025-12-13 17:31:32
Message-ID: CAOj6k6dy2CRVA6Lsb5N59zE-7KNVKt=oYwWyg8ULK8zOOY8e7A@mail.gmail.com
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hello PG Hackers,

Want to submit a patch that implements zstd compression for TOAST data
using a 20-byte TOAST pointer format, directly addressing the concerns
raised in prior discussions [1
<https://www.postgresql.org/message-id/flat/CAFAfj_F4qeRCNCYPk1vgH42fDZpjQWKO%2Bufq3FyoVyUa5AviFA%40mail.gmail.com#e41c78674adfa4d16b2fa82e59faf9aa>
][2
<https://www.postgresql.org/message-id/flat/CAJ7c6TOtAB0z1UrksvGTStNE-herK-43bj22=5xVBg7S4vr5rQ(at)mail(dot)gmail(dot)com>
][3
<https://www.postgresql.org/message-id/flat/YoMiNmkztrslDbNS(at)paquier(dot)xyz>].

A bit of a background in the 2022 thread [3
<https://www.postgresql.org/message-id/flat/YoMiNmkztrslDbNS(at)paquier(dot)xyz>],
Robert Haas suggested:
"we had better reserve the fourth bit pattern for something extensible e.g.
another byte or several to specify the actual method"

i.e. something like:
00 = PGLZ
01 = LZ4
10 = reserved for future emergencies
11 = extended header with additional type byte

Michael also asked whether we should have "something a bit more extensible
for the design of an extensible varlena header."

This patch implements that idea.
The format:

struct varatt_external_extended {
int32 va_rawsize; /* same as legacy */
uint32 va_extinfo; /* cmid=3 signals extended format */
uint8 va_flags; /* feature flags */
uint8 va_data[3]; /* va_data[0] = compression method */
Oid va_valueid; /* same as legacy */
Oid va_toastrelid; /* same as legacy */
};

*A few notes:*

- Zstd only applies to external TOAST, not inline compression. The 2-bit
limit in va_tcinfo stays as-is for inline data, where pglz/lz4 work fine
anyway. Zstd's wins show up on larger values.
- A GUC use_extended_toast_header controls whether pglz/lz4 also use the
20-byte format (defaults to off for compatibility, can enable it if you
want consistency).
- Legacy 16-byte pointers continue to work - we check the vartag to
determine which format to read.

The 4 extra bytes per pointer is negligible for typical TOAST data sizes,
and it gives us room to grow.

Regards,
Dharin

Attachment Content-Type Size
zstd-toast-compression-external.patch application/octet-stream 80.8 KB

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Christoph Berg 2025-12-13 17:36:07 Re: failed NUMA pages inquiry status: Operation not permitted
Previous Message Mihail Nikalayeu 2025-12-13 17:00:00 Re: Issues with ON CONFLICT UPDATE and REINDEX CONCURRENTLY