Re: toast table corrupted by vacuum - missing chunk number 0 for toast value

From: Peter Geoghegan <pg(at)bowt(dot)ie>
To: Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
Cc: Tomas Vondra <tomas(at)vondra(dot)me>, Andrey Borodin <x4mmm(at)yandex-team(dot)ru>, Bruce Momjian <bruce(at)momjian(dot)us>, PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org>, dbryan(dot)green(at)gmail(dot)com
Subject: Re: toast table corrupted by vacuum - missing chunk number 0 for toast value
Date: 2026-08-22 23:59:13
Message-ID: CAH2-WznHNyjerv+kZGvDgf_YNuDAKXB24Z6n1KPr9pUXPPpOnQ@mail.gmail.com
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On Sat, Aug 22, 2026 at 10:25 AM Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com> wrote:
> Unfortunately these backups was overwritten month ago

I thought that your complaint might be related to a recently reported
issue with subxact corruption:

https://postgr.es/m/447185db-3f06-4a38-8518-ecbd42d8b7d5@gmail.com

That case involves a subtransaction that has already subcommitted
being forced to abort. This reminded me of what Tomas said about it
being weird that the TOAST table's entry appears to be for a
committing transaction, yet other evidence suggests that the same XID
aborted.

I asked Opus 5 to build a test case (based on one from Andrey Borodin
on that other thread) that produces symptoms consistent with Pavel's
report. It succeeded in doing so -- see the attached patch adding
these tests.

These tests fail, demonstrating the same symptoms (you might have to
comment-out an assertion in clog.c to allow the script to get as far
as executing the pageinspect steps that show those specific details).
With Bryan Green's bug fix patch from the other thread applied, the
tests all pass.

Of course, I'm not claiming that this is incontrovertible proof that
Pavel's case was an example of this bug causing corruption in the
field; other bugs have had similar symptoms. For example, the snapshot
export bug I'm working on [1] can make committed subxacts temporarily
appear to have aborted, albeit on a standby (that one doesn't fit
Pavel's reported symptoms but isn't too far off). I still think this
explanation has a decent chance of being the correct one.

With that said, Pavel should look at this test case and try to find
any factor that might rule this explanation out. It would be worth
working out which ERROR actually happened. We simulate an OOM or other
error using the test's subxact-after-childxids-transfer injection
point; we don't know what ERROR actually occurred in the field (if we
assume this really was the bug involved, then some specific error must
have triggered it).

Question for Pavel: When I run the failing test, I see "WARNING:
AbortSubTransaction while in COMMIT state" in the test server logs. Do
any of the logs that you collected show that? If they do, then this
diagnosis is almost certainly correct.

[1] https://postgr.es/m/CAH2-WzmHVeYY=pjz9x8DhhxVjXHX0pvoQ-MdiB1Tt6=o2GTiKg@mail.gmail.com
--
Peter Geoghegan

Attachment Content-Type Size
0001-Demonstrate-TOAST-corruption-from-a-subtransaction-a.patch application/octet-stream 15.5 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message David Rowley 2026-08-23 01:38:17 Re: Add bms_offset_members() function for bitshifting Bitmapsets
Previous Message Michael Paquier 2026-08-22 22:34:15 Re: Reduce memory overheads for storing a Memoize tuple