Re: alternative compression algorithms?

Hi,

On 2015-04-19 22:51:53 +0200, Tomas Vondra wrote:
> The reason why I'm asking about this is the multivariate statistics patch -
> while optimizing the planning overhead, I realized that considerable amount
> of time is spent decompressing the statistics (serialized as bytea), and
> using an algorithm with better decompression performance (lz4 comes to mind)
> would help a lot. The statistics may be a few tens/hundreds kB, and in the
> planner every millisecond counts.

I've a hard time believing that a different compression algorithm is
going to be the solution for that. Decompressing, quite possibly
repeatedly, several hundreds of kb during planning isn't going to be
fun. Even with a good, fast, compression algorithm.

But independent of this, it'd be good to make progress on it.

> Would a differentiated approach work? That is, either adding an initdb
> option allowing the user to choose an alternative compression algorithm (and
> thus let him consider the possible patent issues), or using different
> algorithms for different pieces of data (e.g. keep pglz for the user data,
> and lz4 for statistics).
>
> The first option is quite trivial to implement - I already have an
> experimental patch implementing that (attached, but a bit dirty). The second
> option is probably more difficult (we'd have to teach tuple toaster about
> multiple compression algorithms and pass that information somehow). Also,
> I'm not sure it'd make the patent concerns go away ...

Note that I had implemented the "second option" somewhere down the lane
in the [2] you refer to.

> [2] http://www.postgresql.org/message-id/20130614230142.GC19641@awork2.anarazel.de

Greetings,

Andres Freund