Re: [HACKERS] Custom compression methods

From: Andrey Borodin <x4mmm(at)yandex-team(dot)ru>
To: Dilip Kumar <dilipbalaut(at)gmail(dot)com>
Cc: Robert Haas <robertmhaas(at)gmail(dot)com>, Tomas Vondra <tomas(dot)vondra(at)2ndquadrant(dot)com>, Alexander Korotkov <a(dot)korotkov(at)postgrespro(dot)ru>, David Steele <david(at)pgmasters(dot)net>, Ildus Kurbangaliev <i(dot)kurbangaliev(at)gmail(dot)com>, Dmitry Dolgov <9erthalion6(at)gmail(dot)com>, PostgreSQL Developers <pgsql-hackers(at)lists(dot)postgresql(dot)org>
Subject: Re: [HACKERS] Custom compression methods
Date: 2020-12-29 17:48:49
Message-ID: 323B1B01-DA42-419F-A99C-23E2C162D53B@yandex-team.ru
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

> 28 дек. 2020 г., в 11:14, Andrey Borodin <x4mmm(at)yandex-team(dot)ru> написал(а):
>
>> Thanks for the patch. Maybe we can allow setting custom compression
>> methods for wal compression as well.
>
> No, unfortunately, we can't use truly custom methods. Custom compression handlers are WAL-logged. So we can use only static set of hardcoded compression methods.

So, I've made some very basic benchmarks on my machine [0].
With pglz after checkpoint I observe 1146 and 1225 tps.
With lz4 I observe 1485 and 1524 tps.
Without wal_compression I see 1529 tps.

These observations can be explained with plain statement: pglz is bottleneck on my machine, lz4 is not.
While this effect can be reached with other means [1] I believe having lz4 for WAL FPIs would be much more CPU efficient.

FPA lz4 for WAL FPI patch v17. Changes: fixed some frontend issues, added some comments.

Best regards, Andrey Borodin.

[0] https://yadi.sk/d/6y5YiROXQRkoEw
[1] https://www.postgresql.org/message-id/flat/25991595-1848-4178-AA57-872B10309DA2%40yandex-team.ru#e7bb0e048358bcff281011dcf115ad42

Attachment Content-Type Size
v17-0007-Add-Lz4-compression-to-WAL-FPIs.patch application/octet-stream 11.9 KB

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Paul Martinez 2020-12-29 20:26:19 [PATCH] Simplify permission checking logic in user.c
Previous Message Tom Lane 2020-12-29 16:45:03 Re: Let's start using setenv()