Re: Different compression methods for FPI

From: Justin Pryzby <pryzby(at)telsasoft(dot)com>
To: Michael Paquier <michael(at)paquier(dot)xyz>
Cc: Andrey Borodin <x4mmm(at)yandex-team(dot)ru>, pgsql-hackers(at)postgresql(dot)org, Thomas Munro <thomas(dot)munro(at)gmail(dot)com>
Subject: Re: Different compression methods for FPI
Date: 2021-05-25 04:44:45
Message-ID: 20210525044444.GV3676@telsasoft.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On Wed, May 19, 2021 at 06:31:15PM +0900, Michael Paquier wrote:
> I still don't understand why XID consistency has anything to do with
> the compression of FPIs. There is nothing preventing the testing of
> compression of FPIs, and plese note this argument:
> https://www.postgresql.org/message-id/BEF3B1E0-0B31-4F05-8E0A-F681CB918626@yandex-team.ru
>
> For example, I can just revert from my tree 0002 and 0003, and still
> perform tests of the various compression methods. I do agree that we
> are going to need to do something about this problem, but let's drop
> this stuff from the set of patches of this thread and just discuss
> them where they are needed.

They are needed here - that they're included is deliberate. Revert this and
then the tests fail. "Make sure published XIDs are persistent"

time make -C src/test/recovery check
# Failed test 'new xid after restart is greater'

> And you have not replaced BKPIMAGE_IS_COMPRESSED by a PGLZ-equivalent,
> so your patch set is eating more bits for BKPIMAGE_* than it needs

The goal is to support 2+ "methods" (including "none"), which takes 4 bits, so
may as well support 3 methods.

- uncompressed
- pglz
- lz4
- zlib or zstd or ??

This version:
0) repurposes the pre-existing GUC as an enum;
1) saves a bit (until zstd is included);
2) shows the compression in pg_waldump;

To support different compression levels, I think I'd change from an enum to
string and an assign hook, which sets a pair of ints.

--
Justin

Attachment Content-Type Size
v8-0001-Allow-alternate-compression-methods-for-wal_compr.patch text/x-diff 20.5 KB
v8-0002-Run-011_crash_recovery.pl-with-wal_level-minimal.patch text/x-diff 1002 bytes
v8-0003-Make-sure-published-XIDs-are-persistent.patch text/x-diff 7.2 KB
v8-0004-wal_compression_method-default-to-zlib.patch text/x-diff 1.4 KB
v8-0005-re-add-wal_compression_method-lz4.patch text/x-diff 7.1 KB
v8-0006-Default-to-LZ4.patch text/x-diff 2.7 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Bharath Rupireddy 2021-05-25 05:29:37 Re: Refactor "mutually exclusive options" error reporting code in parse_subscription_options
Previous Message Greg Nancarrow 2021-05-25 04:12:30 Re: Re: Parallel scan with SubTransGetTopmostTransaction assert coredump