Re: Different compression methods for FPI

From: Andrey Borodin <x4mmm(at)yandex-team(dot)ru>
To: Michael Paquier <michael(at)paquier(dot)xyz>
Cc: Justin Pryzby <pryzby(at)telsasoft(dot)com>, pgsql-hackers <pgsql-hackers(at)postgresql(dot)org>, Thomas Munro <thomas(dot)munro(at)gmail(dot)com>
Subject: Re: Different compression methods for FPI
Date: 2021-05-31 07:33:44
Message-ID: 3252174C-C21D-4711-A038-5230DE465BAE@yandex-team.ru
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

> 25 мая 2021 г., в 12:26, Michael Paquier <michael(at)paquier(dot)xyz> написал(а):
>
> On Mon, May 24, 2021 at 11:44:45PM -0500, Justin Pryzby wrote:
>> The goal is to support 2+ "methods" (including "none"), which takes 4 bits, so
>> may as well support 3 methods.
>>
>> - uncompressed
>> - pglz
>> - lz4
>> - zlib or zstd or ??
>
> Let's make a proper study of all that and make a choice, the only
> thing I am rather sure of is that pglz is bad compared to all the
> others. There is no point to argue as long as we don't know if any of
> those candidates are suited for the job.

There's a lot of open studies like [0,1].
In short, Lz4 is fastest codec. Zstd gives better compression at cost of more CPU usage[2]. Zlib is not tied to Facebook, however, it's slower and have smaller compression ratio than Zstd. Zstd is actively developed.

There is Google's Brotli codec. It is comparable to Zstd.

What kind of deeper study do we want to do?
Would it make sense to run our own benchmarks?
Or, perhaps, review other codecs?

In my opinion, anything that is sent over network or written to block storage deserves Zstd-5 compression. But milage may vary.

Thanks!

Best regards, Andrey Borodin.

[0] https://indico.cern.ch/event/695984/contributions/2872933/attachments/1590457/2516802/ZSTD_and_ZLIB_Updates_-_January_20186.pdf
[1] https://facebook.github.io/zstd/
[2] Zstd gives significantly better compression at cost of little more CPU usage. But they both have points on Pareto frontier.

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Kyotaro Horiguchi 2021-05-31 07:58:25 Duplicate history file?
Previous Message Esteban Zimanyi 2021-05-31 07:29:42 How to disable the autovacuum ?