Re: [REVIEW] Re: Compression of full-page-writes

On Wed, Dec 17, 2014 at 12:12 AM, Merlin Moncure <mmoncure(at)gmail(dot)com> wrote:

> Compressing *one* file with lz4 and a quick/n/dirty plgz i hacked out
> of the source (borrowing heavily from
> https://github.com/maropu/pglz_bench/blob/master/pglz_bench.cpp), I
> tested the results:
>
> lz4 real time: 0m0.032s
> pglz real time: 0m0.281s
>
> mmoncure(at)mernix2 ~/src/lz4/lz4-r125 $ ls -lh test.*
> -rw-r--r-- 1 mmoncure mmoncure 2.7M Dec 16 09:04 test.lz4
> -rw-r--r-- 1 mmoncure mmoncure 2.5M Dec 16 09:01 test.pglz

> A better test would examine all manner of different xlog files in a
> fashion closer to how your patch would need to compress them but the
> numbers here tell a fairly compelling story: similar compression
> results for around 9x the cpu usage.
>
Yes that could be better... Thanks for some real numbers that's really
informative.

Be advised that compression alg
> selection is one of those types of discussions that tends to spin off
> into outer space; that's not something you have to solve today. Just
> try and make things so that they can be switched out if things
> change....
>
One way to get around that would be a set of hooks to allow people to set
up the compression algorithm they want:
- One for buffer compression
- One for buffer decompression
- Perhaps one to set up the size of the buffer used for
compression/decompression scratch buffer. In the case of pglz, this needs
for example to be PGLZ_MAX_OUTPUT(buffer_size). The part actually tricky is
that as xlogreader.c is used by pg_xlogdump, we cannot directly use a hook
in it, but we may as well be able to live with a fixed maximum size like
BLCKSZ * 2 by default, this would let largely enough room for all kinds of
compression algos IMO...
Regards,
--
Michael