Re: BTree index corruption (heap-tid-past-end, unexpected zero page, misplaced TID in posting list) recurring on high-churn tables, PG 18.3, data_checksums=on, no preceding crash

On Mon, Aug 10, 2026 at 3:44 AM Alessandro Regolini
<alessandro(at)regolini(dot)it> wrote:
> This morning production was failing with
>
> ERROR: heap tid from index tuple (14066,1) points past end of heap page line pointer array at offset 287 of block 7 in index "mpaapp_cache_content_type_id_object_i_f6ecfefa_uniq"

Unfortunately, those symptoms are extremely broad. The underlying
cause could be almost anything.

There are no known bugs in 18.3 that match the symptoms you're
describing -- nothing that seems remotely relevant changed in 18. The
most likely explanation is storage-level corruption, possibly caused
by a broken backup tool.

> Nine minutes after the last of those errors, bt_index_check(heapallindexed => true) passed on all five indexes of that table, and verify_heapam returned zero rows. I had seen the same thing four weeks earlier on another cluster and concluded it had healed itself, which was wrong. heapallindexed checks heap to index, and what I have is the other direction, an index
> tuple too many. So your precondition for sending page images never triggers here.

Right, amcheck can only detect structural inconsistencies within an
index (which sometimes happens to you), and, with heapallindexed,
where an index tuple points to something in the heap whose indexed
columns don't match what is stored in the index. The logic for
heapallindexed piggy-backs on the code that REINDEX uses, and verifies
that a fingerprint of the would-be index has tuples that were present
in the index. As you say, if there are "extra" index tuples in the
index it won't be caught -- heapallindexed effectively assumes those
are dead tuples, that the simulated REINDEX correctly excluded from
its imaginary/fingerprint-only new index.

--
Peter Geoghegan