Re: Zstandard support for toast compression

Greetings,

* Robert Haas (robertmhaas(at)gmail(dot)com) wrote:
> On Tue, May 17, 2022 at 12:19 AM Michael Paquier <michael(at)paquier(dot)xyz> wrote:
> > Toast compression is supported for LZ4, and thanks to the refactoring
> > work done with compression methods assigned to an attribute, adding
> > support for more methods is straight-forward, as long as we don't
> > support more than 4 methods as the compression ID is stored within the
> > first 2 bits of the raw length.
>
> Yeah - I think we had better reserve the fourth bit pattern for
> something extensible e.g. another byte or several to specify the
> actual method, so that we don't have a hard limit of 4 methods. But
> even with such a system, the first 3 methods will always and forever
> be privileged over all others, so we'd better not make the mistake of
> adding something silly as our third algorithm.

In such a situation, would they really end up being properly distinct
when it comes to what our users see..? I wouldn't really think so.

> I don't particularly have anything against adding Zstandard
> compression here, but I wonder whether there's any rush. If we decide
> not to add this now, we can always change our minds and add it later,
> but if we decide to add it now, there's no backing it out. I'd
> probably be inclined to wait and see if our public demands it of us.

If anything, this strikes me as a reason to question using a bit for LZ4
and not a mark against Zstd. Still tho- there seems like a clear path
to having more than 4 when we get demand for it, and here's a patch for
what is pretty clearly one of the better compression methods out there
today. As another point, while pgbackrest supports gzip, lz4, zstd, and
bzip2, where it's supported, zstd seems to be the most used. We had
gzip first as zstd wasn't really a proper thing at the time, and lz4 for
speed. Bzip2 was added more as it was easy to do and of some interest
on systems that didn't have zstd but I wouldn't support adding it to PG
as I'd hope that nearly all systems where v16 is deployed will have Zstd
support.

+1 for adding Zstd for me.

Thanks,

Stephen