Re: refactoring basebackup.c

On Wed, Sep 22, 2021 at 12:41 PM Jeevan Ladhe
<jeevan(dot)ladhe(at)enterprisedb(dot)com> wrote:
> If I set prefs->autoFlush to 0, then LZ4F_compressUpdate() returns an
> error: ERROR_dstMaxSize_tooSmall after a few iterations.
>
> After digging a bit in the source of LZ4F_compressUpdate() in LZ4 repository, I
> see that it throws this error when the destination buffer capacity, which in
> our case is mysink->base.bbs_next->bbs_buffer_length is less than the
> compress bound which it calculates internally by calling LZ4F_compressBound()
> internally for buffered_bytes + input buffer(CHUNK_SIZE in this case). Not sure
> how can we control this.

Uggh. It had been my guess was that the reason why
LZ4F_compressBound() was returning such a large value was because it
had to allow for the possibility of bytes inside of its internal
buffers. But, if the amount of internally buffered data counts against
the argument that you have to pass to LZ4F_compressBound(), then that
makes it more complicated.

Still, there's got to be a simple way to make this work, and it can't
involve setting autoFlush. Like, look at this:

https://github.com/lz4/lz4/blob/dev/examples/frameCompress.c

That uses the same APIs that we're here and a fixed-size input buffer
and a fixed-size output buffer, just as we have here, to compress a
file. And it probably works, because otherwise it likely wouldn't be
in the "examples" directory. And it sets autoFlush to 0.

--
Robert Haas
EDB: http://www.enterprisedb.com