Re: adding 'zstd' as a compression algorithm

On 2/15/22 16:10, Peter Geoghegan wrote:
> On Tue, Feb 15, 2022 at 12:00 PM Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>> I'm not sure I completely follow this. There are cases where we use
>> compression algorithms for internal purposes, and we can change the
>> defaults without users knowing or caring. For example, we could decide
>> that LZ4-compressing FPIs in WAL records is a sensible default and
>> just start doing it, and users need not care.

I'm not sure that this is entirely true. lz4 is available on most
non-EOL distros but you don't need to look to far to find a distro that
does not have it. So, choosing lz4 by default could impact binary
portability. Of course, there are lots of other factors that impact
binary portability, e.g. collations, but this would add one more.

This is even more true for zstd since it is not as ubiquitous as lz4. In
fact, it is not even included with base RHEL8. You need to install EPEL
to get zstd.

>> But backups are
>> different, because when you pg_basebackup -Ft, you get .tar or .tar.gz
>> or .tar.lz4 files which we don't give you tools to extract.
>
> What I meant is that you're buying into an ecosystem by choosing to
> use a tool like pgBackrest. That might not be the right thing to do
> universally, but it comes pretty close. You as a user are largely
> deferring to the maintainers and their choice of defaults. Not
> entirely, of course, but to a significant degree, especially in
> matters like this. There aren't that many dimensions to the problem
> once the question of compatibility is settled (it's settled here
> because you've already bought into a tool that requires the library as
> standard, say).

As much as we would like to, we have not changed the default compression
for pgBackRest. lz4 and zstd are both optional at build time since they
are not always available (though lz4 is getting close). So we have left
gzip as the default, though a lot of that has to do with maintaining
compatibility with prior versions of pgBackRest.

Having said all that, I'm absolutely in favor of adding zstd to Postgres
as an optional compression format.

Regards,
David