Re: libpq compression

On 15 May 2018 at 21:36, Andrew Dunstan <andrew(dot)dunstan(at)2ndquadrant(dot)com>
wrote:

>
>
>>> > To use zstd compression, Postgres should be configured with
>>> --with-zstd. Otherwise compression will use zlib unless it is disabled by
>>> --without-zlib option.
>>> > I have added compression=on/off parameter to connection string and -Z
>>> option to psql and pgbench utilities.
>>>
>>> I'm a bit confused why there was no reply to this. I mean, it wasn't
>>> sent on
>>> 1st April, the patch still can be applied on top of the master branch
>>> and looks
>>> like it even works.
>>>
>>> I assume the main concern her is that it's implemented in a rather not
>>> extensible way. Also, if I understand correctly, it compresses the data
>>> stream
>>> in both direction server <-> client, not sure if there is any value in
>>> compressing what a client sends to a server. But still I'm wondering why
>>> it
>>> didn't start at least a discussion about how it can be implemented. Do I
>>> miss
>>> something?
>>>
>>
>> Implementation of libpq compression will be included in next release of
>> PgProEE.
>> Looks like community is not so interested in this patch. Frankly speaking
>> I do not understand why.
>>
>
I'm definitely very interested, and simply missed the post.

I'll talk with some team mates as we're doing some PG12 planning now.

> Yes, this patch is not extensible: it can use either zlib either zstd.
>> Unfortunately internal Postgres compression pglz doesn't provide streaming
>> API.
>> May be it is good idea to combine it with Ildus patch (custom compression
>> methods): https://commitfest.postgresql.org/18/1294/
>>
>
Given the history of issues with attempting custom/pluggable compression
for toast etc, I really wouldn't want to couple those up.

pglz wouldn't make much sense for protocol compression anyway, except maybe
for fast local links where it was worth a slight compression overhead but
not the cpu needed for gzip. I don't think it's too exciting. zlib/gzip is
likely the sweet spot for the reasonable future for protocol compression,
or a heck of a lot better than what we have, anyway.

We should make sure the protocol part is extensible, but the implementation
doesn't need to be pluggable.

In this case it will be possible to use any custom compression algorithm.
>> But we need to design and implement streaming API for pglz and other
>> compressors.
>>
>
> I'm sure there is plenty of interest in this. However, you guys need to
> understand where we are in the development cycle. We're trying to wrap up
> Postgres 11, which was feature frozen before this patch ever landed. So
> it's material for Postgres 12. That means it will probably need to wait a
> little while before it gets attention. It doesn't mean nobody is interested.
>
> I disagree with Dmitry about compressing in both directions - I can think
> of plenty of good cases where we would want to compress traffic from the
> client.
>

Agreed. The most obvious case being COPY, but there's also big bytea
values, etc.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services