Re: toast table corrupted by vacuum - missing chunk number 0 for toast value

On 8/21/26 17:47, Pavel Stehule wrote:
> Hi
>
> pá 21. 8. 2026 v 12:31 odesílatel Andrey Borodin <x4mmm(at)yandex-team(dot)ru
> <mailto:x4mmm(at)yandex-team(dot)ru>> napsal:
>
> Hi Pavel,
>
> On Thu, Aug 20, 2026 at 9:56 PM Pavel Stehule wrote:
> > We didn't find any way how this issue can be reproduced. We manually
> > executed VACUUM on the database that was restored to the time close
> > before this issue - and the data was not broken - but now, I am not
> > sure what was the minor version that we used for this test.
>
> I wonder whether the missing TOAST data could be a secondary symptom,
> rather than evidence that VACUUM removed TOAST belonging to the current
> row version.
>
>
> I have dumped pages
>
> https://www.postgresql.org/message-id/attachment/199738/
> investigation.txt <https://www.postgresql.org/message-id/
> attachment/199738/investigation.txt>
>
> xmin, xmax was same on main table, and toast table. These rows was not
> updated.
>

Thanks. Interesting. It's weird the TOAST rows have XID 2779428784 and
flags {HEAP_HASVARWIDTH,HEAP_XMIN_COMMITTED,HEAP_XMAX_INVALID}, yet the
VACUUM apparently decided all of them are dead. Judging by:

> rmgr: Heap2 len (rec/tot): 59/ 99, tx: 0, lsn:
> D6E/8A8A6AD0, prev D6E/8A8A6A68, desc: PRUNE latestRemovedXid 0
> nredirected 0 ndead 4, blkref #0: rel 1663/4267629/16352341 blk
17880358 FPW

I mean, ndead=4 seems to suggest that. But how could that be with
HEAP_XMIN_COMMITTED?

I wonder if the FPW from the WAL matches what you saw as corrupted, or
if it's correct (it should, given the ndead=4). But maybe try dumping it
using pg_waldump - although, 14 doesn't have that :-(.

What happened with the XID 2779428784? Did it commit sometime right
before the vacuum, or what? Judging by the page LSN, it must have been
pretty far back.

What does the clog say about this transaction? I think it should be:
pg_xact/0A5A, page 21, byte offset 175596, bits 0-1, so

$ od -An -tx1 -j 175596 -N 1 $PGDATA/pg_xact/0A5A

Also, what does the RUNNING_XACTS right before/after vacuum say?

--
Tomas Vondra