Re: ZSTD TOAST compression, and an extensible compression method encoding

From: Manu <manuelreyesbravo(at)gmail(dot)com>
To: Greg Burd <greg(at)burd(dot)me>
Cc: Nikhil Kumar Veldanda <veldanda(dot)nikhilkumar17(at)gmail(dot)com>, Michael Paquier <michael(at)paquier(dot)xyz>, Japin Li <japinli(at)hotmail(dot)com>, pgsql-hackers(at)lists(dot)postgresql(dot)org
Subject: Re: ZSTD TOAST compression, and an extensible compression method encoding
Date: 2026-10-17 19:48:53
Message-ID: 179226653323.878814.9853291079649992732@gmail.com
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hi Greg,

Thanks for re-running it on your own hosts and folding the reset into 0003 --
much appreciated, and good that the slice/prefix and memory numbers line up.

> I didn't reproduce the silent wrong-data case you saw, only the false error

The silent one needs a prefix that stops the stream mid-frame rather than a
corrupt value read to the end. Attached is what I used (rename back to .sh/.py;
run as BIN=/path/to/install/bin bash repro-dctx-reset.sh). In one backend,
without the ZSTD_DCtx_reset(): read a prefix of a value that ends before its
frame does (check B1), then take a slice of a different value (B2) -- that
slice comes back with wrong bytes and no error, while full reads stay correct
because ZSTD_decompressDCtx() starts a fresh frame. My setup reaches the
mid-frame stop with a 1 MB value whose third block header is corrupted
(corrupt_block.py): the prefix before the bad block decodes fine and leaves
the reused context mid-frame, and the next value's streaming slice then reads
stale state. With the reset in place B2, and all 221 values (md5-checked), are
right. Same root cause as the false error you saw; the reset closes both, and
the silent variant is the one I'd least want a connection pool to hit.

Thanks also for adding me to 0003. One small thing: the authorship trailer did
not make it into the posted v8-0003 commit message -- for the record, my name
there is Manuel Reyes Bravo <manuelreyesbravo(at)gmail(dot)com>.

Regards,
Manu

Attachment Content-Type Size
repro-dctx-reset.sh.txt text/plain 5.8 KB
corrupt_block.py.txt text/plain 1.9 KB

In response to

Browse pgsql-hackers by date

  From Date Subject
Previous Message Greg Burd 2026-10-17 19:28:43 Re: ZSTD TOAST compression, and an extensible compression method encoding