| From: | Manu <manuelreyesbravo(at)gmail(dot)com> |
|---|---|
| To: | Greg Burd <greg(at)burd(dot)me> |
| Cc: | Nikhil Kumar Veldanda <veldanda(dot)nikhilkumar17(at)gmail(dot)com>, Michael Paquier <michael(at)paquier(dot)xyz>, Japin Li <japinli(at)hotmail(dot)com>, pgsql-hackers(at)lists(dot)postgresql(dot)org |
| Subject: | Re: ZSTD TOAST compression, and an extensible compression method encoding |
| Date: | 2026-10-15 19:49:08 |
| Message-ID: | 179209374832.830484.4266062766962664931@gmail.com |
| Views: | Whole Thread | Raw Message | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-hackers |
Hi Greg,
> Given Manu's numbers [...] I'd fold something similar to his change
> into 0003.
Thanks for picking this back up, and for tying it to Michael's
857340b0263 / ce3387b74b7 -- that's the precedent the reuse was waiting
on.
I re-measured the ZSTD_DCtx reuse on v7 (over 54635e6892f). September
holds: on 32 kB values slice and prefix are 0.62-0.75 of v7, full read
0.86-0.87 (json stays ~1.0), random 1 MB slice 0.67, and 4 kB 0.92-0.98.
Compression is unchanged. make check passes (240 tests) in -O2 and under
cassert+ASan.
Attached (nocfbot-v7-0003-con-dctx-reuse.patch.txt) is your 0003 with
the reuse folded in -- Nikhil still the author; against the current 0003
it only touches toast_compression.c, and the commit message notes
Michael's two commits.
The one thing worth checking on a long-lived context is the reset at the
start of each use. I corrupted a block header mid-frame: the decode
fails with "compressed zstd data is corrupt" and every later read is
correct, for full and slice and after a rollback. A mutant without the
reset fails -- after an errored slice every later slice errors, and
after an early-cut prefix another value's slice returns wrong data with
no error. Memory stays ~160 kB for 32 kB slices, at most ~2.5 MiB after
large slices (the default window cap), and zstd shrinks it back itself.
On scope: Michael ties the WAL's decompression context to the
XLogReaderState that owns WAL decoding; detoast has no such owner, so the
backend is the natural scope. Same pattern, scope per subsystem.
This is independent of your 0004 dict_dctx. One note: the comment in
0004 about moving its slice to the streaming loop would need the same
reset -- the mutant above shows why.
Regards,
Manu
| Attachment | Content-Type | Size |
|---|---|---|
| nocfbot-v7-0003-con-dctx-reuse.patch.txt | text/plain | 82.1 KB |
| evidence-dctx-v7.txt | text/plain | 5.1 KB |
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Zsolt Parragi | 2026-10-15 19:54:12 | Parallel TID Range Scan returns wrong results on rescan |
| Previous Message | Tom Lane | 2026-10-15 19:34:03 | Re: [PG19] psql backslash commands fail with translated messages and a non UTF-8 client encoding |