Re: Unexpected behavior after OOM errors

Matthias van de Meent <boekewurm+postgres(at)gmail(dot)com> wrote:

> On Mon, 20 Jul 2026 at 19:48, Antonin Houska <ah(at)cybertec(dot)at> wrote:
> >
> > Michael Paquier <michael(at)paquier(dot)xyz> wrote:
> >
> > > On Fri, Jul 17, 2026 at 12:00:01PM +0300, Alexander Lakhin wrote:
> > > > 4) make_new_heap():
> > > > --- a/src/backend/commands/repack.c
> > > > +++ b/src/backend/commands/repack.c
> > > > @@ -1255,3 +1255,5 @@ make_new_heap(Oid OIDOldHeap, Oid NewTableSpace, Oid NewAccessMethod,
> > > > */
> > > > +oom_prob = 0.01;
> > > > CommandCounterIncrement();
> > > > +oom_prob = 0;
> > > >
> > > > triggers:
> > > > TRAP: failed Assert("ItemIdIsNormal(lp)"), File: "heapam.c", Line: 2813, PID: 943626
> > > > also:
> > > > ERROR: attempted to delete invisible tuple
> > >
> > > I'll let Alvaro comment on this one. It deserves an open item.
> >
> > I was also curious because I worked on repack.c recently. After applying (a
> > bit rebased) random-oom-errors.patch from [1] and after turning some "out of
> > memory" errors to PANIC, I could reproduce the error even on PG 18. The stack
> > trace is below. So it does not seem to be PG 19 specific.
>
> A PANIC with AllocSetAlloc on the stacktrace isn't an indication of a
> bug when you add both an OOM probability on palloc, and set the
> OOM-error-thrower to PANIC instead of ERROR.
>
> This thread contains various examples of backend-local state remaining
> in various levels of incomplete modifications/corruptions even after
> the ERROR handling of an OOM has been completed. PANICs cause the
> backend to be terminated, and so are much less likely to cause
> problems down the line.
>
> Note, too, that the problem that Alexander reported is a failing
> Assert(), and an ERROR reporting "attempted to delete invisible
> tuple", rather than actual "out of memory" ERRORs.

I supposed that these problems happen because relcache remains in wrong state
due to the "out of memory" ERROR. So the point of using PANIC was to find out
when exactly the cache gets broken. However, I missed the fact that
Alexander's patch triggers the ERROR at several different places, so the stack
trace I posted might be unrelated.

Nevertheless, on a closer look I concluded that the problem is not related to
OOM. Rather, *any* ERROR that happens during cache invalidation can cause
this. Attached here is cache_inval_failure.diff that reproduces the problem
reliably:

postgres=# CREATE TABLE t1(i int PRIMARY KEY); CLUSTER t1 USING t1_pkey; DROP TABLE t1;
CREATE TABLE
ERROR: failure during cache invalidation
ERROR: attempted to delete invisible tuple

(The Assert(ItemIdIsNormal(lp)) IMO fires only if VACUUM manages to reset the
item pointer before heap_delete() gets called. With autovacuum=off, I get only
the "failure during cache invalidation" error.)

So I think we need to make sure that ERROR during cache invalidation becomes
FATAL.

--
Antonin Houska
Web: https://www.cybertec-postgresql.com