PostgreSQL 18.4 backend SIGSEGV in pgstat_gc_entry_refs() after caught DSM attach error

Hello,

We had a production PostgreSQL 18.4 background worker segfault in
pgstat_gc_entry_refs(). Analysis of the core with the matching executable
and debug symbols found incomplete backend-local pgstats references and an
orphan DSM descriptor. The same worker had caught and logged
"can't attach the same segment more than once" about 31.5 hours earlier.

An earlier burst of "Too many open files" errors is the likely initiating
resource failure, although I have not yet found a log line that ties EMFILE,
the affected PID, and the DSM handle together. I have separated the facts
from that inference below.

Environment
-----------

PostgreSQL: 18.4, Debian package 18.4-1.pgdg13+1
Architecture: Linux amd64
Container userspace: Debian 13 (trixie), glibc 2.41-12+deb13u3
Host kernel: Linux postgresql.threatview.app 7.0.12+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 7.0.12-2~bpo13+1 (2026-06-21) x86_64 GNU/Linux
dynamic_shared_memory_type: posix
max_connections: 160
shared_buffers: 32205MB
Old RLIMIT_NOFILE: 1024
New RLIMIT_NOFILE: 8192

postgres executable build ID:

8b04ec1cf00c6d647b30cb22c3209516bb028299

postgres executable SHA-256:

0d62d25714729ec59d89638323084d5142ab519126a98e2c0ede97f54423bfbf

The backend was the PoWA 5.1.1 background worker. Relevant installed
extension versions were:

powa 5.1.1
pg_stat_statements 1.12
pg_stat_kcache 2.3.1
pg_qualstats 2.1.3
pg_wait_sampling 1.1

The executor-hook frames from these extensions look like the expected hook
chain. PoWA is relevant because powa_take_snapshot() catches errors from
individual snapshot functions and the worker continues with the same
backend-local state.

Timeline and server log
-----------------------

Approximately two days before the crash there was a group of "Too many open
files" errors. The file-descriptor limits were subsequently raised. The
archived logs have not yet established whether PID 3496738 was already
running when the first of those errors occurred.

At 2026-08-04 01:06:34.375 UTC, PID 3496738 logged this caught error:

WARNING: public.powa_take_snapshot(0):
function public.powa_statements_snapshot failed:
state : XX000
message: can't attach the same segment more than once

The PL/pgSQL context showed the error occurring while
powa_statements_snapshot() was collecting pg_stat_statements data. I can
provide the complete statement and PL/pgSQL context if useful.

At 2026-08-05 08:36:34.452 UTC, the same PID failed:

LOG: background worker "powa" (PID 3496738) was terminated by signal 11: Segmentation fault
DETAIL: Failed process was running: SET search_path TO pg_catalog;SELECT public.powa_take_snapshot()
LOG: terminating any other active server processes
LOG: all server processes terminated; reinitializing

The time between the caught DSM error and the SIGSEGV was 31 hours and
30 minutes.

Resolved backtrace
------------------

The core was analysed using the matching PostgreSQL 18.4 executable and
postgresql-18-dbgsym 18.4-1.pgdg13+1.

The relevant resolved frames are:

#0 pgstat_gc_entry_refs() at pgstat_shmem.c:783
[inlined into pgstat_get_entry_ref()]
#1 pgstat_prep_pending_entry()
#2 pgstat_assoc_relation()
#3 _bt_first()
#4 btgettuple()
#5 index_getnext_tid()
#6 index_getnext_slot()
...
#50 SPI_execute()
#51 powa_main() at powa.c:442
#52 BackgroundWorkerMain()

SIGINFO was:

si_code = SEGV_MAPERR
si_addr = 0x10

The faulting instructions were:

mov 0x18(%rax),%rdx # ent->entry_ref
mov (%rdx),%rcx # entry_ref->shared_entry
cmpb $0,0x10(%rcx) # shared_entry->dropped; fault here

rdx pointed to a valid PgStat_EntryRef, but rcx was zero. Thus this was a
NULL shared_entry dereference, not a fault while decoding an index page.

The entry encountered first by GC was:

key.kind = 1
key.dboid = 4
key.objid = 0

entry_ref->shared_entry = NULL
entry_ref->shared_stats = NULL
entry_ref->generation = 0
entry_ref->pending = NULL

The current operation happened to be associating relation statistics for an
index lookup on powa_snapshot_metas_pkey. I see no evidence of B-tree or
index corruption; the index access merely caused pgstats GC to run.

Backend-local pgstats cache findings
------------------------------------

The backend-local pgstats hash contained 159 live slots. Nine entries had
an allocated PgStat_EntryRef with shared_entry == NULL and
shared_stats == NULL:

kind dboid objid
---- ---------- --------
1 1 0
1 4 0
2 91510651 91513773
2 91510651 91512915
2 91510651 91513057
2 91510651 91512852
2 91510651 91513762
2 91510651 91512074
2 91510651 3542

The fatal database-OID-4 entry was simply the first incomplete entry reached
in iterator order.

DSM findings from the core
--------------------------

dsm_segment_list contained exactly two descriptors. One was the current,
healthy pgstats DSA expansion mapping:

handle = 2349205466
mapped_address = 0x7f9c19042000
mapped_size = 1048576
resowner = NULL

The other descriptor was incomplete:

handle = 3452972756
control_slot = 3
resowner = NULL
mapped_address = NULL
mapped_size = 0

The incomplete descriptor was still linked into dsm_segment_list. Other
session-level DSA candidates inspected in the core were NULL, making the
pgstats DSA the likely owner.

Working analysis
----------------

This section is analysis rather than direct observation.

The pgstats DSA is session-pinned. When DSA lazily maps an expansion
segment, get_segment_by_index() temporarily selects area->resowner (NULL for
a pinned area) and calls dsm_attach().

dsm_attach() creates and links a backend-local descriptor, then increments
the shared control slot's reference count, before calling:

dsm_impl_op(DSM_OP_ATTACH, ..., ERROR)

With POSIX DSM, the implementation then uses shm_open(), fstat(), and mmap().
If one of those operations raises ERROR, dsm_attach() has no local cleanup
around dsm_impl_op(). Because this mapping deliberately has no
ResourceOwner, transaction/subtransaction cleanup does not remove the
descriptor. A retry finds the handle in dsm_segment_list and raises:

can't attach the same segment more than once

The earlier "Too many open files" errors make shm_open(...)=EMFILE a strong
candidate for the first failure. We are still searching archived logs for
handle 3452972756 or /PostgreSQL.3452972756 to confirm that part.

Independently, pgstat_get_entry_ref_cached() inserts and zeroes a
backend-local PgStat_EntryRef before shared dshash/DSA access. An ERROR
during the subsequent lookup or lazy segment mapping therefore leaves an
entry with both shared pointers NULL. A later lookup of the same key can
repair that entry, but pgstat_get_entry_ref() invokes pgstat_gc_entry_refs()
before doing the cache lookup. GC currently dereferences shared_entry
without checking it.

PoWA catches the ERROR and continues in the same process, preserving both
pieces of backend-local state. The nine incomplete references are
consistent with repeated failed snapshot accesses. After a later
gc_request_count change, GC iterated those references and the backend
segfaulted.

This appears related in class to these commits, but neither covers the exact
path seen here:

8191e0c16a0373f851a9f5a8112e3aec105b5276
Fix corruption of pgstats shared hashtable due to OOM failures

16ebc196d8926823bc8d64931557b679cecd67aa
Improve pgstat_get_entry_ref_cached() behavior on OOM

16ebc196 handles failure to allocate entry_ref itself. In this core,
entry_ref was allocated, but an ERROR occurred before its shared fields were
populated.

As of 2026-08-05, I can still see the relevant exception-safety gaps in:

REL_18_STABLE 13a9be148e530b9b5c0af2b499e3a08c1f5eb053
master f585671055d1c56d6fba0bc5835d28e68248ccfe

Reproduction status
-------------------

I have a deterministic source-level TAP test, attached separately as:

postgresql-pgstats-dsm-error-recovery-test.patch

The test adds a fault-injection point immediately before DSM_OP_ATTACH, plus
test-only helpers in src/test/modules/injection_points. It:

* starts a backend that will keep its backend-local pgstats cache alive;
* resolves the test functions before the pgstats DSA expansion segment is
created, so this backend has not yet mapped that DSM segment;
* creates enough fake relation pgstats entries from another backend to force
a DSA allocation in a DSM-backed expansion segment;
* injects ERROR at the lazy DSM attach point in the persistent backend;
* catches the ERROR inside an internal subtransaction;
* requests pgstats entry-reference GC; and
* performs another lookup of the same pgstats entry.

With the RFC fix patch applied, the test passes on Linux amd64:

./configure --without-icu --enable-tap-tests \
--enable-injection-points --enable-cassert
make -j2
make -C src/test/modules/injection_points check \
PROVE_TESTS=t/002_pgstats_dsm.pl PROVE_FLAGS=-v

The passing TAP output includes:

ok 1 - created pgstats entry in DSM-backed DSA segment
ok 2 - pgstats lookup recovers after injected DSM attach ERROR

I also ran a negative control against REL_18_STABLE commit
13a9be148e530b9b5c0af2b499e3a08c1f5eb053 with the bug-fix hunks omitted and
the same injection point inserted into the original unfixed dsm_attach()
body. In that build, the same test crashes the backend during the recovery
query:

# issuing query 5 via background psql:
# EXECUTE pgstats_dsm_recovery('800000000'::oid)
Dubious, test returned 29
Result: FAIL

The test node log records:

LOG: client backend (PID 13120) was terminated by signal 11: Segmentation fault
DETAIL: Failed process was running: EXECUTE pgstats_dsm_recovery('800000000'::oid)

Thus the deterministic test both passes with the proposed fix and reproduces
the SIGSEGV when the fix is absent. The production evidence above still came
from an unmodified release build.

Suggested patch
---------------

I have attached two patches:

postgresql-pgstats-dsm-error-recovery-rfc.patch
postgresql-pgstats-dsm-error-recovery-test.patch

The first is intended as a discussion patch, not a claim that the preferred
fix belongs at exactly these layers. It:

* makes dsm_attach() undo its descriptor/reference after an OS mapping
ERROR;
* completes PgStat_EntryRef fields before releasing the dshash lock;
* removes the local ref on the explicit pgstats DSA-allocation failure;
* makes pgstats GC discard incomplete local references; and
* includes the substance of 16ebc196 for REL_18_STABLE.

The fix patch applies to both the REL_18_4 tag and current REL_18_STABLE. It
passes git diff --check, compiles successfully on REL_18_STABLE, and passes
the attached deterministic fault-injection TAP test. The 16ebc196 hunk is
already present on master and would be omitted there.

The second patch is a companion test patch. It applies after the RFC fix
patch. For the negative-control run described above, I applied the same test
helpers and inserted the DSM attach injection point into the original
unfixed dsm_attach() body, because the companion patch's dsm.c hunk is
written against the RFC patch's PG_TRY block.

I avoided catching arbitrary errors around dshash/DSA work and performing
shared cleanup in the catch block, because an ERROR may have occurred while
a DSA or dshash lock is held. The proposed GC change confines that recovery
to backend-local objects.

Questions
---------

1. Does the analysis of the incomplete DSM descriptor and pgstats references
look correct?
2. Should dsm_attach() itself guarantee cleanup after dsm_impl_op() raises,
or would the DSA caller be the preferred cleanup layer?
3. Is discarding incomplete backend-local references in
pgstat_gc_entry_refs() the right defensive behaviour, or should creation
be refactored to provide stronger exception safety?
4. Should the CurrentResourceOwner changes around DSA attach/create also use
PG_FINALLY so the previous owner is restored on ERROR?
5. If confirmed, which supported branches should receive the fixes?

Source references
-----------------

REL_18_4 dsm_attach():

https://github.com/postgres/postgres/blob/REL_18_4/src/backend/storage/ipc/dsm.c#L665-L748

REL_18_4 DSA lazy segment mapping:

https://github.com/postgres/postgres/blob/REL_18_4/src/backend/utils/mmgr/dsa.c#L1748-L1808

REL_18_4 pgstats cache acquisition and GC:

https://github.com/postgres/postgres/blob/REL_18_4/src/backend/utils/activity/pgstat_shmem.c#L400-L539
https://github.com/postgres/postgres/blob/REL_18_4/src/backend/utils/activity/pgstat_shmem.c#L757-L796

Related commits:

https://github.com/postgres/postgres/commit/8191e0c16a0373f851a9f5a8112e3aec105b5276
https://github.com/postgres/postgres/commit/16ebc196d8926823bc8d64931557b679cecd67aa

I have retained the core, exact executable, library files, and surrounding
logs and can provide additional GDB output. I cannot post the raw core to a
public list because it contains production process memory and potentially
sensitive query/data contents.

Core SHA-256, for evidence tracking:

030f338df8089bbb2befccad7fff8520b112f02dd9cfa728b139219b66103703

Best regards,
Niall Newman

Solutions Architect/Lead Developer
Turaco Labs Ltd.
nn(at)turacolabs(dot)com