Re: zstd compression for pg_dump

Hi!

> On Jan 4, 2021, at 11:04 AM, Andrey Borodin <x4mmm(at)yandex-team(dot)ru> wrote:
>
> Daniil, is levels definition compatible with libpq compression patch?
> +typedef struct Compress {
> + CompressionAlgorithm alg;
> + int level;
> + /* Is a nondefault level set ? This is useful since different compression
> + * methods have different "default" levels. For now we assume the levels
> + * are all integer, though.
> + */
> + bool level_set;
> +} Compress;

Similarly to this patch, it is also possible to define the compression level at the initialization stage in libpq compression patch.

The difference is that in libpq compression patch the default compression level always equal to 1, independently of the chosen compression algorithm.

> On Jan 4, 2021, at 11:04 AM, Andrey Borodin <x4mmm(at)yandex-team(dot)ru> wrote:
>
> Libpq compression encountered some problems with memory consumption which required some extra config efforts.

> On Jan 4, 2021, at 12:06 PM, Justin Pryzby <pryzby(at)telsasoft(dot)com> wrote:
>
> RAM use is not significantly different from zlib, except that zstd --long adds
> more memory.

Regarding ZSTD memory usage:

Recently I’ve made a couple of tests of libpq compression with different ZLIB/ZSTD compression levels which shown that compressing/decompressing ZSTD w/ high compression levels
require to allocate more virtual (Commited_AS) memory, which may be exploited by malicious clients:

https://www.postgresql.org/message-id/62527092-16BD-479F-B503-FA527AF3B0C2%40yandex-team.ru

We can avoid high memory usage by limiting the max window size to 8MB. This should effectively disable the support of compression levels above 19:
https://www.postgresql.org/message-id/6A45DFAA-1682-4EF2-B835-C5F46615EC49%40yandex-team.ru

So maybe it is worthwhile to use similar restrictions in this patch.

—
Daniil Zakhlystov