Re: Compression of bigger WAL records

Hi Michael!

On 8 Oct 2026, at 02:04 UTC, Michael Paquier wrote:
> Applied the recovery part.

Thank you for reviewing, benchmarking and committing it!

Interesting difference in our setups is libzstd: my has 1.5.5, while your
three machines use 1.5.7. I have not checked yet whether that explains the
larger gain here.

Attached is v11 with the three remaining patches, rebased on master:

0001 reuses the compression context on WAL insertion. I restored the
fallback to an uncompressed image if context allocation fails, rather
than risking a PANIC in a critical section.

0002 compresses whole records, allowing lz4 and zstd to share matches
across full-page images. It now reuses the committed reader context too.
The decompression buffer follows the reader's lifetime, and the reader
checks that decompression produces the declared length.

0003 keeps zstd history across records to reach small records that do
not compress well on their own. It is still WIP and off by default.
Reader memory use and the cost of rewinding to a reset boundary remain
the main trade-offs. The decompressors now have context-reset cleanup.

The allocation question from my previous mail is still open: 0002 needs
two buffers in place of the old per-image workspace. I have not found a
way to avoid the extra buffer without complicating the insertion path.

Thank you!

Best regards, Andrey Borodin.