Re: Compression of bigger WAL records

> On 19 Aug 2026, at 11:11, Anthonin Bonnefoy <anthonin(dot)bonnefoy(at)datadoghq(dot)com> wrote:
>
> On a second look, 1.3MB is the size reported by ZSTD_estimateCCtxSize
> with the default compression level for the worst case. So it's the
> maximum memory a cctx can use, and my test was probably not enough to
> reach this.

Hi Anthonin,

Thanks for profiling it, and for noting the 1.3MB mistake.

A compression context sizes itself for the largest input it has been
given. Measured with ZSTD_sizeof_CCtx at the default level:

fresh 5248
after one 8kB page image 154616
after 1000 page images 154616
after one 274300-byte record 1303544
ZSTD_estimateCCtxSize(3) 1303544

So 152kB is what a backend holds if we actually use only step 1.

Where I had thought about context size was 0003, since that is where a
context stays alive across many records and holds a window of history,
and where I measured what a stream costs a reader. What I had not
understood is that the allocation is gradual.

While measuring I also took the decompression side, since 0001 now
keeps one of those too: a DCtx is 95992 bytes and does not grow, which
matches ZSTD_estimateDCtxSize() exactly. So every reader (startup process,
walsender, pg_waldump) pays about 94kB.

> Would it be worth it to register ZSTD_freeCCtx using on_proc_exit? I
> imagine this will be tagged as leaked memory since it's never freed.

I do not think it will be tagged: the context is reachable from a
file-scope static for the life of the process, so a leak checker
classes it as still reachable rather than lost. In basebackup_zstd.c
and astreamer_zstd.c, are tied to the lifetime of a sink or a streamer
rather than of the process.

That said it is one line and I have no strong feeling about it, so if
you or anyone else would rather see it registered, say so and I will
add it.

Thank you!

Best regards, Andrey Borodin.