Re: ZSTD TOAST compression, and an extensible compression method encoding

From: Greg Burd <greg(at)burd(dot)me>
To: Hannu Krosing <hannuk(at)google(dot)com>
Cc: pgsql-hackers(at)lists(dot)postgresql(dot)org, Manu <manuelreyesbravo(at)gmail(dot)com>, Nikhil Kumar Veldanda <veldanda(dot)nikhilkumar17(at)gmail(dot)com>, Michael Paquier <michael(at)paquier(dot)xyz>, Japin Li <japinli(at)hotmail(dot)com>, Dharin Shah <dharinshah95(at)gmail(dot)com>
Subject: Re: ZSTD TOAST compression, and an extensible compression method encoding
Date: 2026-11-21 17:08:09
Message-ID: 179527803395.3315701.16127468513985923668@burd.me
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On Sat, 21 Nov 2026 00:21:49 +0100, Hannu Krosing <hannuk(at)google(dot)com> wrote:
> In this pass I also specifically tested partitioned tables and legacy
> inheritance end-to-end (multi-level declarative partitioning, ATTACH /
> DETACH [CONCURRENTLY] including dropped/reordered columns, cross-partition
> UPDATE / MERGE / INSERT ... SELECT, trigger transition tables, partitioned
> indexes, whole-tree REPACK / VACUUM FULL, pg_upgrade, and pgoutput logical
> replication with publish_via_partition_root = true and REPLICA IDENTITY
> FULL), checking on-disk headers via pageinspect. Dictionaries stay
> strictly attached to individual leaf partitions / tables throughout, with
> zero cross-partition dictionary references.

Hi Hannu,

Thank you. The dropped and reordered columns across ATTACH and DETACH
are a case I had not tried, so that is good to know.

> At some future time it may make sense to re-visit the strict one
> relation == one dictionary mapping and see if we can do better for
> partitioned tables, where same dictionary could be used for many
> partitions.
>
> If nothing else, then maybe just copy the dictionary to the other
> table when UPDATE moves a row there and the dictionary for some fields
> in row does not exist.

Agreed, and I think it belongs in its own thread once this is in.
Copying the dictionary along with a moved row keeps the lifecycle
local: the copy belongs to the target partition and retires with it,
so nothing in this series stands in its way. Sharing one dictionary
across partitions is the bigger step, since it brings back the
question of when a shared dictionary may be dropped.

> (One strictly non-blocking committer-level trivia note in verify_heapam.c,
> not worth a v21 on its own: at chunksize == 4 with method_bits == 3,
> method_byte stays -1 so toast_compression_id_resolve() already reports
> "has invalid compression method id 3 in its data" and returns before
> reaching the `else if (method_bits == VARLENA_COMPRESS_METHOD_LONG)` arm,
> while 5..8-byte truncated dictionary headers in 0004 hit the new
> header_size check as intended -- so every truncated length is caught.)

Right. I forged them on a cluster to be sure: chunk 0 cut to 3 and 4
bytes on a zstd value, and to 4, 5, 8 and 9 bytes on a zstd_dict one.
3 reports the short header, 4 the invalid ID 3, 5 and 8 the 9-byte
header, and 9 passes.

The flip side is that before 0004 the header_size test can never
fire: by the time the method is known, chunk 0 already holds the 4- or
5-byte header. I kept it in 0002 so that 0004 only adds the
dictionary case, but if a committer would rather 0002 not carry a
check that 0004 makes live, it moves over without touching anything
else.

> I have no further findings; the whole v20 series (0001-0005) looks Ready
> to me.

Thanks for staying with this through so many versions. v20 still
applies cleanly to master and cfbot is green on it, so I'm not posting
a new version just yet.

-greg

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Zhijie Hou 2026-11-21 17:09:28 Fix standby logical slot creation during dynamic logical decoding activation
Previous Message Rui Zhao 2026-11-21 17:04:26 Fwd: Fix containment and B-tree ordering for cubes of different dimensions