Re: libpq compression (part 2)

If there's no objection, I'd like to move this to the next CF for consideration
in PG16.

On Mon, Jan 17, 2022 at 10:39:19PM -0600, Justin Pryzby wrote:
> On Tue, Jan 18, 2022 at 02:06:32AM +0500, Daniil Zakhlystov wrote:
> > > => Since March, errmsg doesn't need extra parenthesis around it (e3a87b4).
>
> > I’ve resolved the stuck tests and added zlib support for CI Windows builds to patch 0003-*. Thanks
> > for the suggestion, all tests seem to be OK now, except the macOS one. It won't schedule in Cirrus
> > CI for some reason, but I guess it happens because of my GitHub account limitation.
>
> I don't know about your github account, but it works for cfbot, which is now
> green.
>
> Thanks for implementing zlib for windows. Did you try this with default
> compressions set to lz4 and zstd ?
>
> The thread from 2019 is very long, and starts off with the guidance that
> compression had been implemented at the wrong layer. It looks like this hasn't
> changed since then. secure_read/write are passed as function pointers to the
> ZPQ interface, which then calls back to them to read and flush its compression
> buffers. As I understand, the suggestion was to leave the socket reads and
> writes alone. And then conditionally de/compress buffers after reading /
> before writing from the socket if compression was negotiated.
>
> It's currently done like this
> pq_recvbuf() => secure_read() - when compression is disabled
> pq_recvbuf() => ZPQ => secure_read() - when compression is enabled
>
> Dmitri sent a partial, POC patch which changes the de/compression to happen in
> secure_read/write, which is changed to call ZPQ:
> https://www.postgresql.org/message-id/CA+q6zcUPrssNaRS+FyoBsD-F2stK1Roa-4sAhFOfAjOWLziM4g@mail.gmail.com
> pq_recvbuf() => secure_read() => ZPQ
>
> The same thing is true of the frontend: function pointers to
> pqsecure_read/write are being passed to zpq_create, and then the ZPQ interface
> called instead of the original functions. Those are the functions which read
> using SSL, so they should also handle compression.
>
> That's where SSL is handled, and it seems like the right place to handle
> compression. Have you evaluated that way to do things ?
>
> Konstantin said he put ZPQ at that layer seems to 1) avoid code duplication
> between client/server; and, 2) to allow compression to happen before SSL, to
> allow both (if the admin decides it's okay). But I don't see why compression
> can't happen before sending to SSL, or after reading from it?