Re: BUG #19519: REPACK can fail due to missing chunk for toast value

On Tue, Jul 07, 2026 at 01:51:32AM +0200, Matthias van de Meent wrote:
> On Tue, 7 Jul 2026, 01:19 Michael Paquier, <michael(at)paquier(dot)xyz> wrote:
>
>> On Mon, Jul 06, 2026 at 08:21:41PM +0200, Matthias van de Meent wrote:
>>> So, in effect, this is an issue that's quite similar to the CIC/RIC
>>> bug of PG14<14.4, except that
>>>
>>> 1.) in this case it's SnapshotAny instead of a
>>> registered-but-invisible MVCC snapshot;
>>> 2.) the issue in this case only happens when toast pointers were
>>> cleaned up and then are dereferenced, rather than HOT chains updated
>>> and reclaimed since the scan started; and finally
>>> 3.) this issue dates back a very long time, likely since TOAST, but
>>> defininitely since PG14 introduced the GlobalVisTestFor apis.
>>
>> You mean 042b584c7f7d, revert of d9d076222f5b. That rings a bell.
>>
>
> Indeed
>
>
>>> So I think the only way to fix this is either
>>> 1.) we update every maintenance process that makes use of SnapshotAny
>>> and could access TOAST tables to handle errors caused by missing TOAST
>>> data when the base relation's tuple is RECENTLY_DEAD, and treat those
>>> as "tuple cleanup has already started, we just weren't aware of it
>>> yet, so ignore this tuple" (as I mentioned in [1]), or
>>
>> That seems like an option worth exploring to me, but I doubt that
>> we'll be able to get something that can be backpatched due to how
>> invasive it is. I strongly suspect that it would require at least one
>> table AM change to cope with the fact that we want to let the callers
>> be OK with TOAST chunks missing in some contexts when grabbing a toast
>> slice. That's perhaps for the best if we think about potential
>> regressions, this is scary and old enough that we may still be OK by
>> living without a backpatch.
>>
>
> I suspect that for indexing we're lucky, in that we have full control
> internally over which tuple data get fed into IndexBuildCallback. We can
> make sure only fully detoasted attributes get passed on, and can be
> implemented with some (a lot) of TRY/CATCH work in or around
> FormIndexDatum. That said, I don't think that's very nice, and I'm also a
> bit concerned about leaking things (memory, buffers, ...) in such an
> approach.

I've quickly thought about some try/catch blocks while detoasting, but
I cannot really get how this would be entirely bullet-proof. It seems
to me that we'll live better if we bite the bullet and go for the
invasive change of passing a state across the stack to let detoasting
know that we are dealing with a rewrite.

> For REPACK/VACUUM FULL/CLUSTER, I think we can do something similar, given
> that this code too has a central loop processing the tuple data, though I
> suspect it'll need a bit more work than the reindex one.

While looking at the whole stack, the handling of dead tuples seems to
go through only index_build_range_scan() and
relation_copy_for_cluster() for heapam. A re-check for TOAST values
would be really bad if we have a lot of dead tuples.

>> 2.) we set the xmin of the CI/R backend to the current OldestXmin for
>>> the relation(s) we're currently processing, and so block all cleanup
>>> of the relevant (toast) data for the duration of this statement or
>>> whenever we check for a newer xmin for the relation (whilst ignoring
>>> our current one).
>>
>> Err, I don't think that would be safe anyway? It's rather uncommon in
>> practice in schemas, but we could trigger function calls, like index
>> expressions, domains, whatever, that access data of other tables while
>> processing one table with a SnapshotAny. The CIC revert of 14.4 was
>> about such cases.
>
> True, but in indexes that is forbidden: mislabeling of non-INMUTABLE
> functions causes broken indexes every day, and we can't be expected to make
> it work there. Similarly, for REPACK it shouldn't evaluate any
> non-indexable expressions, right?

Still I am pretty sure that the window would exist, users like fancy
definitions. It seems at least worth looking if we could make
something more stable across the board. Just playing with it now, and
it does not seem that bad in practice, actually..

> Either way, if we can register a snapshot in the REPACK/INDEX backend whose
> xmin is (*correctly*) registered with the OldestNonRemovableTransactionId()
> of the relation, then concurrent cleanup won't happen for toast, and we'd
> also avoid dereferencing those dangling toast pointers. We won't need to
> actually use that snapshot for any scans; it'll avoid the issue as long as
> our backend advertises an xmin <= the OldestXmin used in maintenance to
> hold back pruning/vacuum on the relation.
>
> (we can adjust that xmin every once in a while with a newer
> OldestNonRemovableTransactionId (excl. our own backend) as long as we also
> adjust the OldestXmin cutoff for the SnapshotAny visibility tests, but I
> don't suggest to backpatch that even newer system, it'd add significant
> complications to an already complicated system)

I doubt that anything would be backpatchable, based on how the
discussion is going. I'm OK to be proven wrong.

> ps. sorry for the formatting, I'm writing this on my phone.

No pb. Thanks.
--
Michael