Re: libpq compression

Hi everyone! Thanks for pushing this important topic!

Just my 2 cents.

> 31 окт. 2020 г., в 02:03, Andres Freund <andres(at)anarazel(dot)de> написал(а):
>
>
>>> I think that would also make cross-version handling easier, because a
>>> newer client driver can send the compression request and handle the
>>> error, without needing to reconnect or such.
>>>
>>> Most importantly, I think such a design is basically a necessity to make
>>> connection poolers to work in a sensible way.
>>
>> I do not completely understand the problem with connection pooler.
>> Right now developers of Yandex Odyssey are trying to support libpq
>> compression in their pooler.
>> If them will be faced with some problems, I will definitely address
>> them.
>
> It makes poolers a lot more expensive if they have to decompress and
> then recompress again. It'd be awesome if even the decompression could
> be avoided in at least some cases, but compression is the really
> expensive part. So if a client connects to the pooler with
> compression=zlib and an existing server connection is used, the pooler
> should be able to switch the existing connection to zlib.

The idea of reusing compressed byte ranges is neat and tantalising from technical point of view.
But the price of compression is 1 cpu for 500MB/s (zstd). With a 20Gbps network adapters cost of recompressing all traffic is at most ~4 cores.
Moreover we tend to optimise pooler for the case when it stands on the same host as a DB does.

Despite of this, I believe having message for changing compression method (including turning it off) is a nice thing to have. I can imagine that we may want functions to control compression level of replication depending on what is bottleneck right now: network or CPU.

But I do not understand how we can have FrontEnd message like "Now both FE and BE speak zstd". Some messages can already be in a flight, BE cannot change them already. It looks like both BE and FE can say "Now I'm speaking zstd:6 starting from next byte." if sender knows that correspondent speaks zstd:6, of cause.

Thanks!

Best regards, Andrey Borodin.