Re: [HACKERS] Custom compression methods

On Mon, Mar 01, 2021 at 10:32:23AM +0530, Dilip Kumar wrote:
> On Sun, Feb 28, 2021 at 9:48 AM Justin Pryzby <pryzby(at)telsasoft(dot)com> wrote:
> >
> > On my PC, this new change is causing a test failure:
> >
> > SELECT SUBSTR(f1, 2000, 50) FROM cmdata1;
> > - substr
> > -----------------------------------------------------
> > - 01234567890123456789012345678901234567890123456789
> > -(1 row)
> > -
> > +ERROR: compressed lz4 data is corrupt
>
> The older version of lz4 had this problem that while decompressing
> partial if we don't give the buffer size up to full data length it was
> failing[1] but it is solved in version 1.9.

Thanks. It seems like that explains it.
I think if that's a problem with recent versions, then you'll have to
conditionally disable slicing.
https://packages.debian.org/liblz4-dev

Slicing isn't generally usable if it sometimes makes people's data inaccessible
and gives errors about corruption.

I guess you could make it a compile time test on these constants (I don't know
the necessary version, though)

#define LZ4_VERSION_MAJOR 1 /* for breaking interface changes */
#define LZ4_VERSION_MINOR 7 /* for new (non-breaking) interface capabilities */
#define LZ4_VERSION_RELEASE 1 /* for tweaks, bug-fixes, or development */
#define LZ4_VERSION_NUMBER (LZ4_VERSION_MAJOR *100*100 + LZ4_VERSION_MINOR *100 + LZ4_VERSION_RELEASE)

If the version is too low, either make it #error, or disable slicing.
The OS usual library version infrastructure will make sure the runtime version
is at least the MAJOR+MINOR of the compile time version.

--
Justin