WAL segment file descriptor leak on read errors can PANIC the server

Hi,

A failed WAL read leaks the open WAL segment file descriptor until the
backend exits. Once the fd limit is reached, all later queries in that
backend fail with "Too many open files". With a connection pooler the
leaks add up across clients. There is no warning or log message for
this. Reproducers are at [1] and [2].

It can happen with logical decoding that cancelled decode calls
exhaust the fds, and depending on which open fails first, the backend
dies with a FATAL or the instance restarts with a PANIC. [2] shows a
PANIC.

It can also happen that the leaked fd is on a WAL segment that has
since been removed. The file is unlinked but its space is not freed
until the backend goes away, so pg_wal still looks small while free
space on the disk keeps shrinking. On a small WAL volume I could fill
the disk this way, and the server then PANICed while writing a new WAL
file with no space left, which took down the database instance.

I noticed this while looking at the error paths in pg_walinspect for
the reported issue [3]. I then used Claude Code to check for the same
issue elsewhere, and it helped me find the other affected paths.

The WAL read paths open the segment file as a plain kernel fd. It is
not a virtual fd and not a transient file, and no resource owner owns
it, so nothing in the backend knows it is open. On an error, the
memory holding the WAL reader is freed with its context, but the
segment file it had open stays open. This leak seems to exist in
pg_walinspect functions, logical decoding functions, 2PC WAL read
code, and the WAL summarizer. All of these except the WAL summarizer
are reachable from SQL in simple ways. The walsender does not have
this issue because it closes the file in its own error cleanup. The
startup process is fine too, its reader does not keep a segment open
this way, and its read errors are FATAL anyway.

This is the same leak that commit 91c40548d5 fixed for 2PC by closing
the fd in XLogReaderFree(), but that close only runs on the normal
path, not when an error is thrown before it.

I think the fix is to register a memory context reset callback on the
context the reader is allocated in. If that context is reset or
deleted while a segment file is still open, the callback closes it.
Doing this in XLogReaderAllocate() covers every caller, present and
future, instead of adding an error handler to each one. On HEAD,
XLogReaderFree() unregisters the callback and the reader keeps a
pointer to it in a new field. On PG18 and older the reader struct
cannot grow because of ABI, and there is no
MemoryContextUnregisterResetCallback(), so there the callback stays
registered and its state lives in a small list local to the WAL
reader. I attached a nocfbot prefixed patch for the back branches
doing that.

There are alternative approaches that have some issues. Wrapping each
caller in its own error handler works but is easy to miss in future
callers. Opening the segment as a transient file, so that it gets
closed at transaction abort, adds a rule that a reader cannot outlive
a transaction, which does not work for the logical walsender since it
reads WAL outside any transaction. Tracking the fd with a resource
owner also fixes all callers in one place, but the extensible resource
owner API only exists in PG17 and later, so it cannot be used in the
older branches.

Please find the attached patches (v1 for HEAD and PG19, nocfbot for
PG18). If the approach looks good, I will prepare patches for all the
remaining back branches using the PG18 approach.

Thoughts?

[1]
$ ulimit -n 200

CREATE TABLE t1 (a int);
INSERT INTO t1 VALUES (1);

-- Each iteration errors at end of WAL and leaks one fd
DO $$
BEGIN
FOR i IN 1..200 LOOP
BEGIN
PERFORM * FROM
pg_get_wal_records_info(pg_current_wal_flush_lsn(),
'FFFFFFFF/FFFFFFFF');
EXCEPTION WHEN OTHERS THEN
NULL;
END;
END LOOP;
END $$;

-- Session can no longer open any file
postgres=# SELECT count(*) FROM t1;
ERROR: could not open file "base/5/2691": Too many open files
LINE 1: SELECT count(*) FROM t1;
^
postgres=# CREATE TABLE t2 (a int);
ERROR: could not create file "base/5/16398": Too many open files

-- Shows the fd table full of pg_wal segments
$ lsof -w -p <backend pid> | grep -c pg_wal
193

[2]
$ ulimit -n 200

SELECT pg_create_logical_replication_slot('s', 'test_decoding');
CREATE TABLE foo(a int, b text);
INSERT INTO foo SELECT i, repeat('x', 200) FROM generate_series(1, 300000) i;

-- Each cancelled decode errors mid-read and leaks one fd
SET statement_timeout = '10ms';
SELECT 'SELECT * FROM pg_logical_slot_peek_changes(''s'', NULL, NULL);'
FROM generate_series(1, 200) \gexec

ERROR: canceling statement due to statement timeout
ERROR: canceling statement due to statement timeout
..
..
ERROR: canceling statement due to statement timeout
ERROR: canceling statement due to statement timeout
PANIC: could not open file "pg_logical/snapshots/0-18692A8.snap": Too
many open files
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.

[3] https://postgr.es/m/80E9F0AD-CFC5-4BE5-81DE-D8FE35E10A1C@gmail.com

--
Bharath Rupireddy
Amazon Web Services: https://aws.amazon.com