Re: ZSTD TOAST compression, and an extensible compression method encoding

From: Manu <manuelreyesbravo(at)gmail(dot)com>
To: Greg Burd <greg(at)burd(dot)me>
Cc: Nikhil Kumar Veldanda <veldanda(dot)nikhilkumar17(at)gmail(dot)com>, Michael Paquier <michael(at)paquier(dot)xyz>, Japin Li <japinli(at)hotmail(dot)com>, pgsql-hackers(at)lists(dot)postgresql(dot)org
Subject: Re: ZSTD TOAST compression, and an extensible compression method encoding
Date: 2026-10-15 19:49:08
Message-ID: 179209374832.830484.4266062766962664931@gmail.com
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hi Greg,

> Given Manu's numbers [...] I'd fold something similar to his change
> into 0003.

Thanks for picking this back up, and for tying it to Michael's
857340b0263 / ce3387b74b7 -- that's the precedent the reuse was waiting
on.

I re-measured the ZSTD_DCtx reuse on v7 (over 54635e6892f). September
holds: on 32 kB values slice and prefix are 0.62-0.75 of v7, full read
0.86-0.87 (json stays ~1.0), random 1 MB slice 0.67, and 4 kB 0.92-0.98.
Compression is unchanged. make check passes (240 tests) in -O2 and under
cassert+ASan.

Attached (nocfbot-v7-0003-con-dctx-reuse.patch.txt) is your 0003 with
the reuse folded in -- Nikhil still the author; against the current 0003
it only touches toast_compression.c, and the commit message notes
Michael's two commits.

The one thing worth checking on a long-lived context is the reset at the
start of each use. I corrupted a block header mid-frame: the decode
fails with "compressed zstd data is corrupt" and every later read is
correct, for full and slice and after a rollback. A mutant without the
reset fails -- after an errored slice every later slice errors, and
after an early-cut prefix another value's slice returns wrong data with
no error. Memory stays ~160 kB for 32 kB slices, at most ~2.5 MiB after
large slices (the default window cap), and zstd shrinks it back itself.

On scope: Michael ties the WAL's decompression context to the
XLogReaderState that owns WAL decoding; detoast has no such owner, so the
backend is the natural scope. Same pattern, scope per subsystem.

This is independent of your 0004 dict_dctx. One note: the comment in
0004 about moving its slice to the streaming loop would need the same
reset -- the mutant above shows why.

Regards,
Manu

Attachment Content-Type Size
nocfbot-v7-0003-con-dctx-reuse.patch.txt text/plain 82.1 KB
evidence-dctx-v7.txt text/plain 5.1 KB

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Zsolt Parragi 2026-10-15 19:54:12 Parallel TID Range Scan returns wrong results on rescan
Previous Message Tom Lane 2026-10-15 19:34:03 Re: [PG19] psql backslash commands fail with translated messages and a non UTF-8 client encoding