Using a stock openssl BIO

Hi,

Thomas^WA bad person recently nerdsniped me (with the help of an accidental use
of an SSL connection in a benchmark leading to poor results) into checking what
would be needed to benefit from SSL/TLS hardware acceleration (available with
suitable hardware, OS support (linux and freebsd) and openssl 3). One problem
turns out to be the custom BIO we use.

Which made me look at why we use those.

In the backend the first reason is:

* Private substitute BIO: this does the sending and receiving using send() and
* recv() instead. This is so that we can enable and disable interrupts
* just while calling recv(). We cannot have interrupts occurring while
* the bulk of OpenSSL runs, because it uses malloc() and possibly other
* non-reentrant libc facilities.

I think this part has been obsolete for a while now (primarily [1]). These days
we always operate the backend sockets in nonblocking mode, and handle blocking
at a higher level. Which is where we then can handle interrupts etc correctly
(which we couldn't really before, because it still wasn't ok to jump out of
openssl).

The second part is
* We also need to call send() and recv()
* directly so it gets passed through the socket/signals layer on Win32.

And the not stated need to set/unset pgwin32_noblock around the recv/send
calls.

I don't think the signal handling stuff is still needed with nonblocking
sockets? It seems we could just ensure that there's a pgwin32_poll_signals()
somewhere higher up in secure_read/write()? E.g. in
ProcessClientReadInterrupt()/ProcessClientWriteInterrupt() or with an explicit
call.

And the pgwin32_noblock handling could just be done outside the SSL_read/write().

On the client side it looks like things would be a bit harder. The main problem
seems to be dealing with SIGPIPE. We could possibly deal with that by moving
the handling of that a layer up. That's a thick nest of ugly stuff :(.

A problem shared by FE & BE openssl is that openssl internally uses
BIO_set_data() inside the BIO routines we reuse. Oops. I hacked around that
using BIO_set_callback_arg()/BIO_get_callback_arg(), but that's not
particularly pretty either, although it probably is more reliable, since
callbacks are intended to be set separately from the BIO implementation.

A better approach might be to move the code using per-bio custom data from
pqsecure_raw_read() up to pqsecure_read() etc.

If we wanted to be able to use TLS acceleration while continuing to use our
custom socket routines we'd have to copy a good bit more functionality from
openssl into our BIO implementations:(.

FWIW, I don't think hardware tls acceleration is a particularly crucial thing
for now. Outside of backups it's rare to have the symmetric encryption part of
tls be the problem these days thanks, to the AES etc functions in most of the
common CPUs.

I don't plan to work on this, but Thomas encouraged me to mention this on the
list when I mention it to him.

Greetings,

Andres Freund

[1] https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=387da18874afa17156ee3af63766f17efb53c4b9