Re: Compression of bigger WAL records

> On 26 Jul 2026, at 21:09, Andrey Borodin <x4mmm(at)yandex-team(dot)ru> wrote:
>
> This is v7

Hi hackers!

This is v8. Still three patches. 0003 now answers the question I left
open in v7 - a reader can start at an arbitrary LSN. 0001 grew to
cover decompression as well, which turned out to be the bigger win, and
running the test suites with the feature actually turned on found a
number of bugs.

Numbers are from two machines, 4-core for 0002 and the compression side
of 0001, 16-core for 0003 and the redo figures, both built with -O2 and
without assertions, fsync off, shared_buffers 8GB. Each was repeated,
where a number moved between runs I say so.

---- 0001: reuse zstd contexts, both directions ----

Still the piece that is useful on its own, and it grew since v7: it now
keeps the decompression context too, which turns out to matter more.

XLogCompressBackupBlock() calls ZSTD_compress() and RestoreBlockImage()
calls ZSTD_decompress(). Both create and destroy a context per call. At
the default level ZSTD_estimateCCtxSize() reports 1.3MB for the
contex. So zstd pays an allocation per full-page image on the
way in, and one per image again on the way out. The patch creates each
on first use and keeps it, the compressor for the life of the backend,
the decompressor in XLogReaderState.

The reading side is where this shows, because one startup process
replays every image. Redoing 200MB of page images:

zstd, master 4.24s zstd, patched 2.33s
lz4, master 1.59s lz4, patched 1.59s

A sequential scan that sets hint bits on a freshly checkpointed table
with wal_log_hints on runs about 26% faster.

This also bears on the "WAL compression setting after PostgreSQL LZ4
default change" thread, where the order zstd -> lz4 -> pglz is proposed
for what "on" should mean. On master, replaying zstd-compressed images
costs 2.7x what lz4 costs, even though zstd wrote 38% less WAL; with the
context kept, that gap falls to 1.5x.

Cons: a backend that used zstd once holds the 1.3MB until it exits, and
a reader holds a decompression context, which is far smaller.

---- 0002: whole-record compression alongside FPI compression ----

Unchanged in design from v7, plus the documentation that was missing.

When a record is larger than wal_compression_threshold it is compressed
as a single unit rather than each full-page image separately, which wins
whenever the images in one record share content.

On CREATE INDEX over 10M random doubles, with the generator seeded so
the runs are comparable:

zstd 144.8MB -> 116.9MB (-19%)
lz4 189.5MB -> 161.0MB (-15%)

Pros: it reaches redundancy that per-FPI compression cannot see, records
below the threshold are untouched, and every record still decodes on its
own, so nothing about how WAL is read changes. Setting the threshold
above the largest possible record restores today's behaviour.

Cons, both unchanged from v7:

1. Memory. A backend with compression enabled holds two 274300-byte
buffers where it used to hold one array of about the same total size:
the staging buffer reproduces the old allocation, the output buffer
is new. The compressed length has to be known before WAL space is
reserved, so the output has to be materialized somewhere.

2. A low threshold can make WAL bigger, because whole-record compression
displaces per-FPI compression for the records it takes and adds
header to each.

---- 0003: compress records against earlier records (WIP) ----

Still WIP, but no longer blocked on the reader problem.

The motivation is unchanged: half the WAL a pgbench run produces is out
of reach for per-record compression, because the median record is far
too small to compress on its own but compresses well against the records
before it.

What is new in v8 is how a reader starts in the middle of WAL.

Every stream starts over at fixed 4MB boundaries. The writer enforces
that rather than hoping for it: a record that would continue a stream
past the next boundary is refused its reserved position and built again
against a stream that starts over. The check is one comparison against
CurrBytePos under the spinlock that reservation already holds; the
boundary is converted to a byte position outside the lock. It fires
about once per stream per 4MB, and I could not measure its cost.

A reader that wants to start at some LSN then rewinds to the boundary
below it and reads forward, which rebuilds the decompressors, and stops
short of the record it was asked for: feeding a record to its
decompressor twice would leave it in a state its successors were not
compressed against. That is XLogBeginReadStreamed(), and pg_waldump,
pg_walinspect, logical decoding, walsummarizer and pg_rewind all use it.
A record whose stream has not been seen to start over refuses to
decompress rather than decoding whatever the bytes happen to mean.

Why a fixed distance and not the WAL segment: how far a reader rewinds
should not change when a cluster is initialised with a different segment
size, and 64MB segments are not unusual. 4MB divides both 16MB and 64MB
segments, so a boundary is also always a page start. Compression is
insensitive to the value - pgbench emits the same WAL per transaction
to within 2% anywhere between 1MB and 64MB - so it is chosen for the
readers.

Numbers below are from a 16-core machine, pgbench scale 100, fsync off
so that this measures the feature and not the disk. WAL bytes per
transaction and throughput, with the stream count matched to the client
count:

clients streams off streams = clients
1 1720 / 2456 1086 / 2340 -37% WAL, -5% tps
8 701 / 18432 530 / 17326 -24% WAL, -6% tps
32 544 / 43729 433 / 44111 -20% WAL, no cost
64 542 / 46173 448 / 39499 -17% WAL, -12% tps

The 32- and 64-client rows answer the question I could not answer in v7:
the stream lease, which is held across compression and insertion, does
not show up as contention when the insert path is already busy. At 32
clients with 32 streams there is no measurable cost at all.

What does show up is that the stream count has to track concurrency.
Eight streams buy 4% at 32 clients and nothing at 64:

64 clients, streams 0 / 8 / 64 -> 542 / 534 / 448 bytes per txn

Building the third patch with wal_compression_streams = 0 reproduces the
second patch's numbers, so the cost is in using the feature, not in
carrying it.

Two other shapes, WAL volume only:

wide UPDATE of 500k rows 223.9MB -> 154.6MB (-31%)
COPY of 3M rows 65.4MB -> 68.6MB (+5%)

The COPY case is the honest counterexample: those records are large and
already compress well on their own, so the stream adds header and buys
nothing. I do not think that argues against the feature, but it does
argue that turning it on should stay a choice.

Memory, peak RSS with 64 clients writing and then a full pg_waldump over
what they wrote:

streams backend peak pg_waldump peak
0 146.6 MB 3.4 MB
8 150.5 MB 9.0 MB
64 150.5 MB 51.5 MB

The writing side costs about 4MB per backend and does not grow with the
stream count, because a backend keeps one compressor rather than one per
stream. The reading side costs about 0.75MB per stream, and every
reader pays it: the startup process, a walsender doing logical decoding,
pg_waldump.

Costs, as I see them:

1. Throughput, when streams are pushed as high as the client count: 12%
at 64 clients on 16 cores. At and below one stream per core I could
not measure a cost.

2. Memory. 0.75MB per stream for every reader, as above.

3. WAL retention. A replication slot has to keep the WAL back to the
reset boundary below what it needs itself, so up to 4MB more.

The same rewind costs reading, not just retention: a reader that
starts in the middle re-reads up to 4MB to rebuild the decompressors,
and it does so whether or not the WAL it is about to read holds any
streams at all. For readers that start once that is nothing; for
walsummarizer, which starts afresh per summary file, it is up to 4MB
per 16MB summarized. Making the rewind happen only when a record
actually turns out to need it is the obvious answer and I have not
got it working yet.

4. Records that someone reads by LSN without replaying what precedes
them have to stay out of the scheme: the checkpoint records including
XLOG_CHECKPOINT_REDO, XLOG_END_OF_RECOVERY, XLOG_SWITCH and PREPARE.

I could not measure the fsync=on case usefully. The disk I have caps at
82MB/s and repeats of one configuration differed by a factor of two, so
I have no throughput claim there; WAL volume did reproduce, 15-25%
lower with streams, in line with the numbers above.

What I would still like opinions on: whether refusing a reserved
position is an acceptable thing for an insertion path to do, and how to
resolve the tension the numbers above show: the ratio wants roughly one
stream per writing backend, while each stream costs every reader
0.75MB.
wal_compression_streams currently caps at 64, which is already generous
for a reader and far short of the backend count on a busy server.

WDYT?

Best regards, Andrey Borodin.