Re: libpq compression

From: Daniil Zakhlystov <usernamedt(at)yandex-team(dot)ru>
To: Robert Haas <robertmhaas(at)gmail(dot)com>, Konstantin Knizhnik <k(dot)knizhnik(at)postgrespro(dot)ru>
Cc: Denis Smirnov <sd(at)arenadata(dot)io>, Andrey Borodin <x4mmm(at)yandex-team(dot)ru>, PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org>
Subject: Re: libpq compression
Date: 2020-12-22 11:24:05
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers


I’ve fixed an issue with compression level parsing in this PR

Also, did a couple of pgbenchmarks to measure database resource usage with different compression levels.

Firstly, I measured the bidirectional compression scenario, i.e. database had to do both compression and decompression:

Database setup:
pgbench "host=xxx dbname=xxx port=5432 user=xxx” -i -s 500

Test run:
pgbench "host=xxx dbname=xxx port=5432 user=xxx compression=zstd:(1/3/5/7/9/11/13/15/17/19/20)" --builtin tpcb-like -t 50 --jobs=64 --client=700

When using bidirectional compression, Postgres resource usage correlates with the selected compression level. For example, here is the Postgresql application memory usage:

No compression - 1.2 GiB

zstd:1 - 1.4 GiB
zstd:7 - 4.0 GiB
zstd:13 - 17.7 GiB
zstd:19 - 56.3 GiB
zstd:20 - 109.8 GiB - did not succeed
zstd:21, zstd:22 > 140 GiB
Postgres process crashes (out of memory)

zlib:1 - 1.35 GiB
zlib:5 - 1.35 GiB
zlib:9 - 1.35 GiB

Full report with CPU/Memory/Network consumption graph is available here:

Then, I’ve disabled the compression for the backend and decompression for the frontend
and measured the resource usage for single-directional compression scenario (frontend compression only, backend decompression only):

For all ZSTD compression levels, database host resource usage was roughly the same, except the Committed Memory (Committed_AS):

no compression - 44.4 GiB
zstd:1 - 45.0 GiB
zstd:3 - 46.1 GiB
zstd:5 - 46.1 GiB
zstd:7 - 46.0 GiB
zstd:9 - 46.0 GiB
zstd:11 - 47.4 GiB
zstd:13 - 47.4 GiB
zstd:15 - 47.4 GiB
zstd:17 - 50.3 GiB
zstd:19 - 50.1 GiB
zstd:20 - 66.8 GiB
zstd:21 - 88.7 GiB
zstd:22 - 123.9 GiB

For all ZLIB compression level, database host resource usage was roughly the same.

Full report with CPU/Memory/Network consumption graph is available here:

To sum up, there is actually almost no difference when decompressing the different compression levels, except the Committed_AS size.

Daniil Zakhlystov

In response to


Browse pgsql-hackers by date

  From Date Subject
Next Message Peter Smith 2020-12-22 11:28:21 Re: Single transaction in the tablesync worker?
Previous Message Hou, Zhijie 2020-12-22 11:23:48 RE: Consider Parallelism While Planning For REFRESH MATERIALIZED VIEW