Re: Compression of bigger WAL records

On Tue, Oct 06, 2026 at 06:09:38PM +0500, Andrey Borodin wrote:
> The reader now owns a ZSTD_DCtx pointer under USE_ZSTD. Backend builds
> register a reset callback in the reader's memory context, and
> XLogReaderFree() unregisters it before freeing the context. I also
> shortened the compression-side comment.

The declaration of zstd.h is actually quite invasive. Another
possibility would be a forward declaration, like:
typedef struct ZSTD_DCtx_s ZSTD_DCtx;

That gives type enforcement, but perhaps you had the right idea in v9
where we used a void* pointer. I'd still keep that behind a USE_ZSTD,
though, with a zstd-ish naming.

I have also been benchmarking the replay path a bit with something
like the following based on pgbench_accounts, that gives a fixed
number of compressible pages, then run series of VACUUM FULL to
generate WAL mostly made of full page records:
CREATE TABLE t (aid int, bid int, abalance int, filler char(84))
WITH (autovacuum_enabled = off);
INSERT INTO t SELECT i, i / 100000 + 1, 0, ''
FROM generate_series(1, 610000) i;
VACUUM t;
CHECKPOINT;
-- series 10k pages (see via pg_waldump -b).
VACUUM FULL t;

On replay (no fsync, -O2, no asserts, data on tmpfs), I am getting
depending on two different hosts a gain of 1.5% and 3.7% in replay
time (100 VACUUM FULLs, so 1M pages), which is not as much as you have
reported, still that's above noise. For 50 lines of extra code, I'll
take that.

I would be interested in any scripts you may have, and attempt to
reproduce the numbers you have posted. I'll also try some more runs
on a third host (much better CPU and disk that I have around,
locally, not in the cloud).

> We could also avoid this cleanup bookkeeping with a static context, as
> in 0002. For now we only decompress independent images in one call, with
> no history to preserve between records.

With xlogreader.c/h being a pluggable facility for frontend and
backend, storing a static state is not really cool IMO.

> My reasoning was that the writer can still store an uncompressed image,
> whereas replay cannot restore a compressed image without the context.
> I changed this as you suggested, but I still think falling back to an
> uncompressed image is preferable. We are in a critical section.
>
> wal_compression can change during a session, so preparing the workspace
> outside critical sections is not straightforward. This is also the
> hardest and most contentious part of the whole-record and streaming
> compression patches. I'm trying to reuse the FPI compression buffers,
> but still need twice as much buffer space as before. I haven't found a
> better approach. Your thoughts on the allocation strategy would be very
> helpful, even without reviewing those patches.

Hmm. OK, perhaps just storing it uncompressed is better if the
context allocation fails, that's unlikely, still perhaps your
alternative is just better than a plain crash. Please let me ponder
on this part a bit more.

I am attaching a slighly-edited v4 for the time being, which is fine
to me code-wise. I am still planning to do a bit more benchmarking
with replay times, but my first impression is that the numbers seem
good.

So, any thoughts or comments?
--
Michael