Re: Corrupt btree index includes rows that don't match

On Fri, Jul 4, 2025 at 9:49 AM Erik Johnston <erikj(at)element(dot)io> wrote:

> Hi, a quick update:
>
> - We have discovered that the corruption was present from before libicu
> update.
> - We ran `pg_amcheck --index state_groups_state_type_idx --heapallindexed
> matrix`, which returned nothing
> - We believe that means that (and matches what we see sampling) the index
> has gained extra entries, i.e. that for a given state group it does return
> all the relevant rows in the table *plus* extra rows.
>
> We are also seeing old state groups starting to point at rows that have
> only just been inserted. For example, querying for 353864583 on the primary
> it returns that row plus four rows that have been inserted today, but on
> the backup from last week an index only scan for 353864583 only returns one
> row. This makes it feel like the corruption is ongoing? Nothing should have
> modified that state group in the interim (they are generally immutable).
>
> This naively feels like when inserting a new row we sometimes add the row
> to the index twice: once pointing from the correct state group to the new
> row, and once from an old state group to the new row?
>
>
Are checksums enabled in the instance?

--
Death to <Redacted>, and butter sauce.
Don't boil me, I'm still alive.
<Redacted> lobster!