Re: QuickLZ compression algorithm (Re: Inclusion in the PostgreSQL backend for toasting rows)

On Mon, Jan 5, 2009 at 2:02 PM, Gregory Stark <stark(at)enterprisedb(dot)com> wrote:
>> Regardless of whether we do that or not, no one has offered any
>> justification of the arbitrary decision not to compress columns >1MB,
>
> Er, yes, there was discussion before the change, for instance:
>
> http://archives.postgresql.org/pgsql-hackers/2007-08/msg00082.php

OK, maybe I'm missing something, but I don't see anywhere in that
email where it suggests NEVER compressing anything above 1MB. It
suggests some more nuanced things which are quite different.

> And do you have any response to this point?
>
> I think the right value for this setting is going to depend on the
> environment. If the system is starved for cpu cycles then you won't want to
> compress large data. If it's starved for i/o bandwidth but has spare cpu
> cycles then you will.
>
> http://archives.postgresql.org/pgsql-hackers/2009-01/msg00074.php

I think it is a good point, to the extent that compression is an
option that people choose in order to improve performance. I'm not
really convinced that this is the case, but I haven't seen much
evidence on either side of the question.

> Well the original code had a threshold above which we *always* compresed even
> if it saved only a single byte.

I certainly don't think that's right either.

...Robert