Re: libpq compression

On 22.06.2018 00:34, Nico Williams wrote:
> On Thu, Jun 21, 2018 at 10:12:17AM +0300, Konstantin Knizhnik wrote:
>> On 20.06.2018 23:34, Robbie Harwood wrote:
>>> Konstantin Knizhnik <k(dot)knizhnik(at)postgrespro(dot)ru> writes:
>>> Well, that's a design decision you've made. You could put lengths on
>>> chunks that are sent out - then you'd know exactly how much is needed.
>>> (For instance, 4 bytes of network-order length followed by a complete
>>> payload.) Then you'd absolutely know whether you have enough to
>>> decompress or not.
>> Do you really suggest to send extra header for each chunk of data?
>> Please notice that chunk can be as small as one message: dozen of bytes
>> because libpq is used for client-server communication with request-reply
>> pattern.
> You must have lengths, yes, otherwise you're saying that the chosen
> compression mechanism must itself provide framing.
>
> I'm not that familiar with compression APIs and formats, but looking at
> RFC1950 (zlib) for example I see no framing.
>
> So I think you just have to have lengths.
>
> Now, this being about compression, I understand that you might now want
> to have 4-byte lengths, especially given that most messages will be
> under 8KB. So use a varint encoding for the lengths.
>
> Nico
No explicit framing and lengths are needed in case of using streaming
compression.
There can be certainly some kind of frames inside compression protocol
itself, but it is intrinsic of compression algorithm.

--

Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company