Re: Unexpected behavior after OOM errors

On Fri, 17 Jul 2026 at 11:00, Alexander Lakhin <exclusion(at)gmail(dot)com> wrote:
>
> Hello Michael and Matthias,
>
> Could you please look at other similar anomalies I have discovered?:
>
> (Assuming the attached OOM injection patch applied.)
> 3) pgstat_prep_pending_entry() / pgstat_write_statsfiles():
> --- a/src/backend/utils/activity/pgstat.c
> +++ b/src/backend/utils/activity/pgstat.c
> @@ -1333,3 +1333,5 @@ pgstat_prep_pending_entry(PgStat_Kind kind, Oid dboid, uint64 objid, bool *creat
>
> +oom_prob = 0.005;
> entry_ref->pending = MemoryContextAllocZero(pgStatPendingContext, entrysize);
> +oom_prob = 0;
> dlist_push_tail(&pgStatPending, &entry_ref->pending_node);
>
> for i in {1..100}; do
> pg_ctl -l server.log start
> echo "
> CREATE TABLE t (a int);
> ALTER TABLE t SET (parallel_workers = 20);
> SET parallel_setup_cost = 0;
> SELECT SUM(a) FROM t;
> DROP TABLE t;
> " | psql
> pg_ctl -l server.log stop
>
> grep 'TRAP' server.log && break;
> done
>
> triggers
> TRAP: failed Assert("!ps->dropped"), File: "pgstat.c", Line: 1730, PID: 844901
> maybe this Assert can be removed, because there is no corresponding
> assert in pgstat_build_snapshot(), but I saw also:
> TRAP: failed Assert("rel->pgstat_info->relation == NULL"), File: "pgstat_relation.c", Line: 142, PID: 157820
> TRAP: failed Assert("ent->entry_ref != NULL"), File: "pgstat_shmem.c", Line: 827, PID: 190431

After instrumenting MemoryContextAllocationFailure with an
errbacktrace() in its ereport(), I found that the
Assert("!ps->dropped") seems to be triggered exclusively by errors
flushing stats in the pgstats shutdown hook in earlier terminated
backends. This can happen because the call stack
pgstat_shutdown_hook()->pgstat_report_stat(true)->pgstat_update_dbstats(now)
can allocate a pending entry if it didn't already have one, and only
after the call to pgstat_update_dbstats() we'll start handling and
unlinking the still-referenced pgstat entries.

I'm having some trouble figuring out what to do to fix it.
ERRORs during shutdown hooks elevate to FATAL, but the FATAL mechanism
assumes the backend will do a clean detach from shmem, which it can't
do if there's an error in the shutdown hook.

It's not clear whether before_shmem_exit()/on_shmem_exit() callbacks
are allowed to (try to) allocate memory, let alone whether it's OK for
them to raise errors: even if errors in that path may be handled in a
reasonably safe manner, I don't think they're expected or allowed to
raise.

I believe adjusting the error level of ERROR to PANIC during the shmem
exit callback handling would fix the Assert() failure caused by the
(leaked) shared pgstat entries with dropped=true, at the cost of
having to restart the cluster on any errors in shmem shutdown hooks.
See attached.

Note: The attached has other issues, though: The PANIC doesn't seem to
cause the server to fully restart in some cases in the provided test
script... I'm not sure if it's this script or a more general issue
with PANIC handling.

Kind regards,

Matthias van de Meent
Databricks (https://www.databricks.com)