Fixes for compression options of pg_receivewal and refactoring of backup_compression.{c,h}

From: Michael Paquier <michael(at)paquier(dot)xyz>
To: Postgres hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org>
Cc: Robert Haas <robertmhaas(at)gmail(dot)com>, Georgios <gkokolatos(at)protonmail(dot)com>
Subject: Fixes for compression options of pg_receivewal and refactoring of backup_compression.{c,h}
Date: 2022-04-11 06:52:08
Message-ID: YlPQGNAAa04raObK@paquier.xyz
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hi all,
(Added Robert and Georgios in CC:)

Since babbbb5 and the introduction of LZ4, I have reworked the way
compression is controlled for pg_receivewal, with two options:
- --compress-method, settable to "gzip", "none" or "lz4".
- --compress, to pass down a compression level, where the allowed
range is 1-9. If passing down 0, we'd get an error rather than
implying no compression, contrary to what we did in ~14.

I initially thought that this was fine as-is, but then Robert and
others have worked on client/server compression for pg_basebackup,
introducing a much better design with centralized APIs where one can
use METHOD:DETAIL for as compression value, where DETAIL is a
comma-separated list of keyword=value (keyword = "level" or
"workers"), with centralized checks and an extensible design.

This is something I think we had better fix before beta1, because now
we have binaries that use an inconsistent set of options. So,
attached is a patch set aimed at rework this option set from the
ground, taking advantage of the recent work done by Robert and others
for pg_basebackup:
- 0001 is a simple rename of backup_compression.{c,h} to
compression.{c,h}, removing anything related to base backups from
that. One extra reason behind this renaming is that I would like to
use this infra for pg_dump, but that's material for 16~.
- 0002 removes WalCompressionMethod, replacing it by
pg_compress_algorithm as these are the same enums. Robert complained
about the confusion that WalCompressionMethod could lead to as this
could be used for the compression of data, and not only WAL. I have
renamed some variables to be more consistent, while on it.
- 0003 is the actual option rework for pg_receivewal. This takes
advantage of 0001, leading to the result of removing --compress-method
and replacing it with --compress, taking care of the backward
compatibility problems for an integer value, aka 0 implies no
compression and val > 0 implies gzip. One bonus reason to switch to
that is that this would make the addition of zstd for pg_receivewal
easier in the future.

I am going to add an open item for this stuff. Comments or thoughts?

Thanks,
--
Michael

Attachment Content-Type Size
v1-0001-Rename-backup_compression.-c-h-to-compression.-c-.patch text/x-diff 32.5 KB
v1-0002-Replace-WalCompressionMethod-by-common-pg_compres.patch text/x-diff 24.2 KB
v1-0003-Rework-compression-options-of-pg_receivewal.patch text/x-diff 12.6 KB

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Michael Paquier 2022-04-11 06:58:01 Re: Outdated copyright year in parse_jsontable.c
Previous Message Thomas Munro 2022-04-11 06:48:58 Re: pgsql: Add TAP test for archive_cleanup_command and recovery_end_comman