Re: Teach pg_receivewal to use lz4 compression

From: Michael Paquier <michael(at)paquier(dot)xyz>
To: gkokolatos(at)pm(dot)me
Cc: Robert Haas <robertmhaas(at)gmail(dot)com>, Magnus Hagander <magnus(at)hagander(dot)net>, Dilip Kumar <dilipbalaut(at)gmail(dot)com>, PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org>
Subject: Re: Teach pg_receivewal to use lz4 compression
Date: 2021-11-04 08:21:53
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Thu, Nov 04, 2021 at 04:31:48PM +0900, Michael Paquier wrote:
> I would have expected read_size to be (readend - readp) to match with
> the remaining data in the read buffer that we still need to read.
> Shouldn't out_size also be LZ4_CHUNK_SZ to match with the size of the
> output buffer where all the contents are read? By setting it to 1, I
> think that this is doing more loops into LZ4F_decompress() than really
> necessary. It would not hurt either to memset(0) those buffers before
> they are used, IMO. I am not completely sure either, but should we
> use the number of bytes returned by LZ4F_decompress() as a hint for
> the follow-up loops?
> +#ifdef HAVE_LIBLZ4
> + while (readp < readend)
> + {
> + size_t read_size = 1;
> + size_t out_size = 1;
> +
> + status = LZ4F_decompress(ctx, outbuf, &out_size,
> + readbuf, &read_size, NULL);

And... It happens that the error from v9 is here, where we need to
read the amount of remaining data from "readp", and not "readbuf" :)

It is already late here, I'll continue on this stuff tomorrow, but
this looks rather committable overall.

Attachment Content-Type Size
v11-0001-Teach-pg_receivewal-to-use-LZ4-compression.patch text/x-diff 20.6 KB

In response to


Browse pgsql-hackers by date

  From Date Subject
Next Message Takashi Menjo 2021-11-04 08:46:18 Re: Map WAL segment files on PMEM as WAL buffers
Previous Message Amit Kapila 2021-11-04 08:10:48 Re: Data is copied twice when specifying both child and parent table in publication