Re: [HACKERS] Custom compression methods (mac+lz4.h)

On Sun, Mar 21, 2021 at 07:11:50PM -0400, Tom Lane wrote:
> I wrote:
> > Justin Pryzby <pryzby(at)telsasoft(dot)com> writes:
> >> On Sun, Mar 21, 2021 at 04:32:31PM -0400, Tom Lane wrote:
> >>> This seems somewhat repeatable (three identical failures in three
> >>> attempts). Not sure why I did not see it yesterday; but anyway,
> >>> there is something wrong with partial detoasting for LZ4.
>
> >> With what version of LZ4 ?
>
> > RHEL8's, which is
> > lz4-1.8.3-2.el8.x86_64
>
> I hate to be the bearer of bad news, but this suggests that
> LZ4_decompress_safe_partial is seriously broken in 1.9.2
> as well:
>
> https://github.com/lz4/lz4/issues/783

Ouch

> Maybe we cannot rely on that function for a few more years yet.
>
> Also, I don't really understand why this code:
>
> /* slice decompression not supported prior to 1.8.3 */
> if (LZ4_versionNumber() < 10803)
> return lz4_decompress_datum(value);
>
> It seems likely to me that we'd get a flat out build failure
> from library versions lacking LZ4_decompress_safe_partial,
> and thus that this run-time test is dead code and we should
> better be using a configure probe if we intend to allow old
> liblz4 versions. Though that might be moot.

The function existed before 1.8.3, but didn't handle slicing.
https://github.com/lz4/lz4/releases/tag/v1.8.3
|Finally, an existing function, LZ4_decompress_safe_partial(), has been enhanced to make it possible to decompress only the beginning of an LZ4 block, up to a specified number of bytes. Partial decoding can be useful to save CPU time and memory, when the objective is to extract a limited portion from a larger block.

Possibly we could allow v >= 1.9.3 || (ver >= 1.8.3 && ver < 1.9.2).

Or maybe not: the second half apparently worked "by accident", and we shouldn't
need to have intimate knowledge of someone else's patchlevel releases,

--
Justin