Re: adding 'zstd' as a compression algorithm

Hi,

On 2022-02-15 11:40:20 -0800, Peter Geoghegan wrote:
> On Tue, Feb 15, 2022 at 10:20 AM Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> > Likewise, I still download the .tar.gz version of anything
> > that gives me that option, basically because I'm familiar with the
> > format and it's easy for me to just carry on using it -- and in a
> > similar way I expect a lot of people will be happy to continue to
> > compress backups with gzip for many years to come.

There's a difference between downloading a source tarball of 1.5x the size,
and 3x the decompression cost (made up numbers), because the cost of either is
not a relevant factor. I think basebackups are a different story.

> Isn't it an incontrovertible fact that LZ4 is superior to pglz in
> every way? LZ4 is pretty much its successor. And so it seems totally
> fine to assume that users will always want to use the clearly better
> option, and that that option will be generally available going
> forward. TOAST compression is applied selectively already, based on
> various obscure implementation details, some of which are quite
> arbitrary.

Yea, we should really default to lz4 in initdb when available. Expecting users
to know about magic incantation to get often vastly superior performance isn't
a good approach. And it even makes initdb itself faster, because it turns out
we compress enough that the cpu cycles for it are a measurable factor.

> I know less about zstd, but I imagine that a similar dynamic exists
> there. Overall, everything that you've said makes sense to me.

IMO zstd has pretty much won the "gzip" type usecase of compression. Even
prior lz4 uses have even been converted to it because it's often cheap enough.

Greetings,

Andres Freund