Re: libpq compression (part 2)

From: Daniil Zakhlystov <usernamedt(at)yandex-team(dot)ru>
To: Justin Pryzby <pryzby(at)telsasoft(dot)com>
Cc: PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org>, Andrey Borodin <x4mmm(at)yandex-team(dot)ru>
Subject: Re: libpq compression (part 2)
Date: 2022-03-03 08:50:21
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Ok, thanks

> On 3 Mar 2022, at 02:33, Justin Pryzby <pryzby(at)telsasoft(dot)com> wrote:
> If there's no objection, I'd like to move this to the next CF for consideration
> in PG16.
> On Mon, Jan 17, 2022 at 10:39:19PM -0600, Justin Pryzby wrote:
>> On Tue, Jan 18, 2022 at 02:06:32AM +0500, Daniil Zakhlystov wrote:
>>>> => Since March, errmsg doesn't need extra parenthesis around it (e3a87b4).
>>> I’ve resolved the stuck tests and added zlib support for CI Windows builds to patch 0003-*. Thanks
>>> for the suggestion, all tests seem to be OK now, except the macOS one. It won't schedule in Cirrus
>>> CI for some reason, but I guess it happens because of my GitHub account limitation.
>> I don't know about your github account, but it works for cfbot, which is now
>> green.
>> Thanks for implementing zlib for windows. Did you try this with default
>> compressions set to lz4 and zstd ?
>> The thread from 2019 is very long, and starts off with the guidance that
>> compression had been implemented at the wrong layer. It looks like this hasn't
>> changed since then. secure_read/write are passed as function pointers to the
>> ZPQ interface, which then calls back to them to read and flush its compression
>> buffers. As I understand, the suggestion was to leave the socket reads and
>> writes alone. And then conditionally de/compress buffers after reading /
>> before writing from the socket if compression was negotiated.
>> It's currently done like this
>> pq_recvbuf() => secure_read() - when compression is disabled
>> pq_recvbuf() => ZPQ => secure_read() - when compression is enabled
>> Dmitri sent a partial, POC patch which changes the de/compression to happen in
>> secure_read/write, which is changed to call ZPQ:
>> pq_recvbuf() => secure_read() => ZPQ
>> The same thing is true of the frontend: function pointers to
>> pqsecure_read/write are being passed to zpq_create, and then the ZPQ interface
>> called instead of the original functions. Those are the functions which read
>> using SSL, so they should also handle compression.
>> That's where SSL is handled, and it seems like the right place to handle
>> compression. Have you evaluated that way to do things ?
>> Konstantin said he put ZPQ at that layer seems to 1) avoid code duplication
>> between client/server; and, 2) to allow compression to happen before SSL, to
>> allow both (if the admin decides it's okay). But I don't see why compression
>> can't happen before sending to SSL, or after reading from it?

In response to


Browse pgsql-hackers by date

  From Date Subject
Next Message Peter Eisentraut 2022-03-03 09:10:34 Re: Support for grabbing multiple consecutive values with nextval()
Previous Message Amit Kapila 2022-03-03 08:41:22 Re: Skipping logical replication transactions on subscriber side