RE: BUG #19411: libpq 16.x exhibits a memory leak when connections are repeatedly created and destroyed

> Tom Lane wrote:
>> Interesting. I wondered about GSSAPI, but spinning up such an
>> environment is more work than I wanted to do on speculation.

Your suspicion was right, and I'm sorry for pointing at psycopg2 earlier. The leak is in libpq, and it needs GSSAPI *authentication*. Every connection to my server authenticates via "gss" in pg_hba.conf. My gssencmode=disable runs only turned off GSS encryption, so none of my tests ever removed it.

pg_GSS_continue() runs once per GSS round trip, and Kerberos with mutual authentication takes two. On every call it does

/* Only try to acquire credentials if GSS delegation isn't disabled. */
if (!pg_GSS_have_cred_cache(&conn->gcred))
conn->gcred = GSS_C_NO_CREDENTIAL;

pg_GSS_have_cred_cache() calls gss_acquire_cred() and stores the result in *cred_out. It doesn't release the credential that is already there. pqDropConnection() releases only the last one, so the credential from the first round trip is orphaned. With MIT krb5 that costs about 17.8 kB per connection, mostly 4144-byte keytab buffers allocated under gss_acquire_cred().

This is gdb on libpq 16.7 with gssencmode=disable, breaking on the GSSAPI entry points. Every connection looks like this:

PQconnectdb
gss_acquire_cred from pg_GSS_have_cred_cache
gss_init_sec_context cred=...b580 from pg_GSS_continue
gss_acquire_cred from pg_GSS_have_cred_cache
gss_init_sec_context cred=...85c0 from pg_GSS_continue
PQfinish
gss_release_cred cred=...85c0 from pqDropConnection

...b580 is never released.

The other callers already guard against this. For example, fe-secure-gssapi.c calls pg_GSS_have_cred_cache() only when conn->gcred == GSS_C_NO_CREDENTIAL. The patch below does the same in pg_GSS_continue(). It's against 18.3 and applies to 16.7 with only a line offset:

--- a/src/interfaces/libpq/fe-auth.c
+++ b/src/interfaces/libpq/fe-auth.c
@@ -101,8 +101,13 @@
if (conn->Pfdebug)
pqTraceOutputMessage(conn, conn->inBuffer + conn->inStart, false);

- /* Only try to acquire credentials if GSS delegation isn't disabled. */
- if (!pg_GSS_have_cred_cache(&conn->gcred))
+ /*
+ * Acquire credentials on the first call only. Each GSS round trip comes
+ * through here, and acquiring again would leak the credential already in
+ * conn->gcred.
+ */
+ if (conn->gcred == GSS_C_NO_CREDENTIAL &&
+ !pg_GSS_have_cred_cache(&conn->gcred))
conn->gcred = GSS_C_NO_CREDENTIAL;

if (conn->gssdelegation && conn->gssdelegation[0] == '1')

I built libpq from the 16.7 and 18.3 release tarballs, with MIT krb5 1.20.1 and OpenSSL 1.1, both with and without the patch. I then ran the reproducer below against a server using gss auth + SSL. These are the bytes per connection still allocated after warm-up, over 450 connections:

unpatched patched
16.7 17823 123
18.3 17844 98

With gssencmode=disable, the patched builds measure 31 and 74 bytes/connection. Connections still authenticate normally. With the patch, gdb shows one gss_acquire_cred() per connection, used for both gss_init_sec_context() calls and released in pqDropConnection(). Over 3,000 connections (least-squares slope after warm-up), the patched builds still grow by 25-31 bytes per connection. libpq 13.11 on the same server grows by 18 bytes per connection, so that remainder doesn't look new in v16.

The unpatched leak also shows up in libpq 17.6 as bundled in psycopg-binary 3.2.13, with MIT krb5 1.15.1: 17985 bytes/connection. libpq 13.11 and 15 don't have it, which fits this code arriving with GSS credential delegation in v16. I haven't tested master.

Andres: thanks for the heaptrack pointer. An allocation tracker with backtraces put 95% of the retained memory under gss_acquire_cred() called from libpq, which is what led here.

--- gss_leak_repro.c ---
/*
* Build: cc -o gss_leak_repro gss_leak_repro.c -I$(pg_config --includedir) \
* -L$(pg_config --libdir) -lpq
* Run: kinit; ./gss_leak_repro "host=<server with gss in pg_hba.conf> dbname=postgres" 1000
* Needs glibc >= 2.33 for mallinfo2().
*/
#include <malloc.h>
#include <stdio.h>
#include <stdlib.h>
#include <libpq-fe.h>

static size_t
heap_in_use(void)
{
struct mallinfo2 mi = mallinfo2();

return mi.uordblks + mi.hblkhd;
}

int
main(int argc, char **argv)
{
const char *conninfo = argc > 1 ? argv[1] : "dbname=postgres";
int n = argc > 2 ? atoi(argv[2]) : 1000;
int warmup = n / 10;
size_t base = 0;

printf("libpq %d\n", PQlibVersion());
for (int i = 1; i <= n; i++)
{
PGconn *conn = PQconnectdb(conninfo);

if (PQstatus(conn) != CONNECTION_OK)
{
fprintf(stderr, "%s", PQerrorMessage(conn));
return 1;
}
if (i == 1)
printf("GSSAPI auth used: %d\n", PQconnectionUsedGSSAPI(conn));
PQfinish(conn);
if (i == warmup)
base = heap_in_use();
}
printf("%.1f bytes per connection still allocated (%d connections after warm-up)\n",
(double) (heap_in_use() - base) / (n - warmup), n - warmup);
return 0;
}
--- end ---

Matt