| From: | Matthew Mochalkin <matthew(dot)mochalkin(at)gmail(dot)com> |
|---|---|
| To: | pgsql-hackers(at)lists(dot)postgresql(dot)org |
| Subject: | libpq: SCRAM client proofs serialize across threads on OpenSSL 3 (re-init of the HMAC per PBKDF2 iteration) |
| Date: | 2026-11-16 21:00:44 |
| Message-ID: | 0B007497-ADC0-43F7-8BE4-BD13F24AA5F2@gmail.com |
| Views: | Whole Thread | Raw Message | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-hackers |
Hi,
A multithreaded libpq client on OpenSSL 3 cannot open SCRAM-SHA-256 connections in parallel. The threads of one process get about the throughput of a single thread, and adding threads makes it worse. The same client as separate processes scales as expected. I ran into it with FrankenPHP (PHP's ZTS build served from the threads of one process, libpq 17.12, OpenSSL 3.5.7, Debian 13): 8 threads doing one connect per request reach ~600 connects/s against ~1,700 for 8 processes, and ~2,900 for the same 8 threads once the role uses md5 or trust.
The cost is in scram_SaltedPassword(). It calls pg_hmac_init() for every one of the 4,096 iterations, and on OpenSSL 3 each HMAC_Init_ex(ctx, key, len, EVP_sha256(), NULL) does an implicit fetch of the digest, which takes process-wide locks inside libcrypto (openssl/openssl#19612 describes the same cost for EVP_DigestInit_ex). With one thread that is a per-iteration overhead; with several threads it is a lock convoy. The EVP_MAC conversion in 1f3b9bb109 keeps the per-iteration EVP_MAC_init() with the digest parameter and has the same behaviour.
A 60-line C program with no libpq in it shows the effect. It runs the 4,096-iteration HMAC-SHA-256 loop in N threads or N forked processes, 8 CPUs, loops per second:
pattern per iteration 1 thr 8 thr 16 thr 8 proc
HMAC_Init_ex(ctx, key, len, EVP_sha256(), NULL) 519 547 252 3,397 (libpq <= 17)
EVP_MAC_init(ctx, key, len, {digest=SHA256}) 600 1,191 547 4,582 (master)
HMAC_Init_ex(ctx, key, len, md_fetched_once, NULL) 1,150 5,684 9,832 6,366
HMAC_Init_ex(ctx, NULL, 0, NULL, NULL) 1,997 9,272 12,095 18,557 (reuse key and digest)
EVP_MAC_init(ctx, NULL, 0, NULL) 1,085 8,045 13,785 13,613
And with libpq itself, PQconnectdb()+PQfinish() in a loop against PostgreSQL 16, scram role:
role 1 thr 8 thr 16 thr 1 proc 8 proc 16 proc
scram-sha-256 237 592 268 222 1,641 2,386
trust 595 1,882 3,814 559 2,640 3,594
This was discussed in February 2025 ("Optimize scram_SaltedPassword performance", https://www.postgresql.org/message-id/Z6B7QXlPwPXewaQZ%40paquier.xyz) That proposal was about single-thread latency and was dropped, partly because a faster PBKDF2 does not help the server much and partly because of the static state in the patch. The multithreaded case is different: it is not that the client is 4 times slower than it could be, it is that N threads are no faster than one, which for a threaded application server means the database connect rate of the whole process is capped around a few hundred per second. The fix does not need any static state: inside scram_SaltedPassword() the key is the same for every iteration, so the OpenSSL implementation can keep it in the context after the first pg_hmac_init() and re-init with a NULL key (HMAC_Init_ex(ctx, NULL, 0, NULL, NULL), or EVP_MAC_init(ctx, NULL, 0, NULL) on the EVP_MAC path). The fallback src/common/hmac.c can keep its current behaviour, or skip re-deriving the pads when the key pointer and length are unchanged.
I can send a patch for the OpenSSL path if there is interest. The reproduction (Docker, PHP and C) is at https://github.com/mattleads/frankenphp-pgconnect-repro.
Regards,
Matt Mochalkin
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Bharath Rupireddy | 2026-11-16 21:41:39 | Re: pg_xmin_horizon: a system view of everything pinning the xmin horizon |
| Previous Message | Jim Jones | 2026-11-16 20:06:46 | Re: ISBN range table |