Re: toast table corrupted by vacuum - missing chunk number 0 for toast value

From: Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
To: Andrey Borodin <x4mmm(at)yandex-team(dot)ru>
Cc: Bruce Momjian <bruce(at)momjian(dot)us>, PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org>
Subject: Re: toast table corrupted by vacuum - missing chunk number 0 for toast value
Date: 2026-08-21 15:47:49
Message-ID: CAFj8pRCiPvuk3jftDaM=jSyjAtL5cK+020=XM36gsd27qQ2o3A@mail.gmail.com
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hi

pá 21. 8. 2026 v 12:31 odesílatel Andrey Borodin <x4mmm(at)yandex-team(dot)ru>
napsal:

> Hi Pavel,
>
> On Thu, Aug 20, 2026 at 9:56 PM Pavel Stehule wrote:
> > We didn't find any way how this issue can be reproduced. We manually
> > executed VACUUM on the database that was restored to the time close
> > before this issue - and the data was not broken - but now, I am not
> > sure what was the minor version that we used for this test.
>
> I wonder whether the missing TOAST data could be a secondary symptom,
> rather than evidence that VACUUM removed TOAST belonging to the current
> row version.
>

I have dumped pages

https://www.postgresql.org/message-id/attachment/199738/investigation.txt

xmin, xmax was same on main table, and toast table. These rows was not
updated.

>
> I have seen incidents where an old row version became visible again.
> Its TOAST data had already been removed legitimately, while the actual
> current row version, including its TOAST data, could still be found by
> primary key. Reading the resurrected version then failed with the same
> missing-chunk error. As far as I recall, in at least one case the old
> tuple was considered visible because of its hint bits even though the
> transaction status did not agree.
>

I don't think this is the same issue. rows in main table was visible by seq
scan

>
> The investigation attached to your original report selects the damaged
> tuple directly by ctid. Could you check whether another version of the
> same logical row exists, in particular one with the same primary key?
> It would be useful to select only ctid, xmin, xmax, and the primary key,
> so that inspecting the rows does not itself try to fetch TOAST data. I
> would also try it with index and bitmap scans disabled.
>

I didn't check this - but we used amcheck, and we didn't find any broken
indexes

>
> The HEAP_UPDATED and HEAP_XMIN_COMMITTED flags on the reported tuple
> make this possibility seem worth checking. If another version exists,
> the pruning of the TOAST page may have been correct, and the actual
> corruption may be that an old heap tuple became visible again. If the
> relevant pg_xact data is still available in the restored instance, it
> would also be interesting to compare the inserting transaction's status
> with the hint bits on this tuple.
>

Unfortunately now it is too late and backups are rewritten :-/. So all
what I have is attached

https://www.postgresql.org/message-id/CAFj8pRDcenNE3qUf%3D6YsrhMLki7BbbM8fLMi9DUzLn_jbLtpGw%40mail.gmail.com

Regards

Pavel

> Thank you!
>
> Best regards, Andrey Borodin.
>
>

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Jim Vanns 2026-08-21 15:56:11 Re: [PATCH] Add support for SAOP in the optimizer for partial index paths
Previous Message Pavlo Golub 2026-08-21 15:42:26 [PATCH v4] Add pg_current_vxact_id() function