Re: ExecForceStoreHeapTuple() loses tts_tid, so ORDER BY-op index scans project an invalid ctid

> On Sep 9, 2026, at 8:18 AM, Greg Burd <greg(at)burd(dot)me> wrote:
>
>
> On Tue, Sep 8, 2026, at 8:03 PM, Michael Paquier wrote:
>> On Tue, Sep 08, 2026 at 01:28:22PM -0400, Greg Burd wrote:
>>> ExecForceStoreHeapTuple() does not set slot->tts_tid when the target
>>> slot is a TTS_IS_BUFFERTUPLE slot. Any plan that re-stores a heap tuple
>>> through it and then projects ctid therefore gets (4294967295,0) instead
>>> of the row's real heap TID.
>>
>> Oops.
>
> Yeah, oops indeed. :)
>
>> My question would be what kind of testing you have done to spot that..
>
> Honestly, not by a test. I am working on a new index AM for vector
> similarity search [1] adding a 1-bit encoding with rerank. To do that
> rerank it sets xs_recheckorderby = true, because its distances are
> quantised and only the executor's exact re-check can order the top-k
> correctly. That routes tuples through nodeIndexscan.c's reorder queue.
> One of our documented recipes harvests ctid from a scan and chains it
> downstream; that quietly started matching nothing, with the ctids
> coming back as (4294967295,0).
>
> My first assumption was that I'd broken something, so I went looking for
> the boundary:
>
> - xs_heaptid was correct and every real column was right; only the
> projected ctid was wrong, which pointed at the slot, not the AM.
> - ExecForceStoreHeapTuple's TTS_IS_BUFFERTUPLE branch calls
> ExecClearTuple (hence ItemPointerSetInvalid on tts_tid) and never
> restores it, while the sibling tts_heap_store_tuple does; and
> slot_getsysattr answers SelfItemPointerAttributeNumber straight out of
> tts_tid.
> - Then I reproduced it with core GiST and nothing else loaded, which is
> the version in the patch.
>
> To confirm the fix I built stock 18.4 with only that hunk applied and
> ran a test:
> unpatched patched
> ctid self-join, expect 5 1 5
> UPDATE ... WHERE ctid, expect 5 1 5
> sentinel ctids at LIMIT 50 49/50 0/50
>
> The UPDATE line is the one that bothers me most. No error, it just
> affects the wrong number of rows.
>
> As for why the tree doesn't catch it, AFAICT nothing in core projects
> ctid from an ORDER BY-op index scan. The GiST kNN tests check ordering
> and results, which are fine here the row data is never wrong. That's
> what v1 adds a regress case for, and I checked it fails
> (ctid_matches = 1) without the hunk and passes (5) with it, so it gates
> the fix rather than just recording current output.
>
> One detail that probably explains the longevity, IndexNextWithReorder
> only queues a tuple when the AM's advertised ORDER BY value doesn't
> compare equal to the recomputed one. GiST's bounding-box distance is
> sometimes exact, so some rows keep their real ctid. Note the 49 of 50
> above rather than 50. My index AM can't usefully bound its distance and
> advertises -inf, so every tuple goes through the queue and every one
> shows the sentinel. Partial in core, total out here.
>
>>> ExecClearTuple() reaches tts_buffer_heap_clear(), which does
>>> ItemPointerSetInvalid(&slot->tts_tid). The tuple is then copied in, but
>>> tts_tid is left invalid. The sibling path, ExecStoreHeapTuple() ->
>>> tts_heap_store_tuple() — *does* slot->tts_tid = tuple->t_self, so this
>>> reads as a plain asymmetry rather than an intentional choice.
>>
>> That's strange. Once thing that I can see why scanning this file is
>> the same code pattern in tts_buffer_heap_copyslot(), where a slot is
>> similarly cleared in a copy-paste fashion.
>
> Agreed.
>
>> Andres?
>> --
>> Michael
>
> best, thanks for looking,
>
> -greg
>
> [1] https://codeberg.org/gregburd/pg_turbovec https://github.com/gburd/pg_turbovec

As can happen it turns out I'm not the first to report this or propose a patch for
it. [1] So I suggest we continue on that thread.

Also, the other report found that the FOR UPDATE can extend the relation and
leave a block that later breaks seqscans so essentially this bug can cause
on-disk damage.

best.

-greg

[1] https://postgr.es/m/CAM6Zo8wZOLnCWRO_tuuXVX9J4N4JN6GsEnk8WJtT0%3D_0zy-1dw%40mail.gmail.com