Re: Possible Visibility Map corruption in supported branches?

Hi,

On 2026-06-26 12:19:55 +0200, Jehan-Guillaume de Rorthais wrote:
> The cluster is running the unsupported 13.23 without checksum enabled, but I
> suspect they would not have helped anyway.
>
> I'm reaching pgsql-hackers because last minor version is from November 2025 and
> unless I'm wrong, no bug related to VM has been fixed in current supported
> releases since then, so they might be subject to the same bug as well (if this
> is a bug). The v19 might not be subject to this though, as Melanie Plageman
> refactored some code around VM there.

There is:

2025-12-16 e4b1986b989 WAL-log inplace update before revealing it to other sessions.

Which could be related. You say there was no crash though in the relevant
timeframe, so maybe the change does not turn out to be relevant.

A few questions to look into:

- You seem to have concluded that the corruption is in the VM, rather than the
heap. Why? You say that is improbable, that's not my experience. I've seen
this kind of thing due to both postgres bugs, OS bugs and storage system
issues.

- How large is the relation, what range of blocks was affected? If it's the
tail of the relation that was affected autovacuum truncations would be where
I'd start looking.

I think it may be the tail, based on :

> INFO: "<TABLE_NAME>": found 524506 removable, 10300212 nonremovable row versions in 489069 out of 3919859 pages
and
> All theses rows came from the blocks 3691935 to 3694483 (about 20MB).

- I'd check the WAL that you collected (as described further down in the
thread) for truncations of the relation. I'd also check if one of the
affected pages has multiple +INIT records.

- Are there any vacuum related WAL records for the affected heap blocks? If
e.g. the relevant rows were removed as part of a VACUUM, this would likely
be evidence of the write somehow having gotten lost.

I'd probably pick one of the affected pages and look for all WAL records
referencing it. That often can tell you most of what happened.

One interesting thing to look at is whether there are WAL records for the
affected page with a later LSN than what you saw when the first vacuum
failed.

- Do you have base backups between the time the rows were inserted and the
time the problem first became apparent? I guess so, based on your later
mail.

I'd check a few of the base backups for
a) the size of the heap relation
b) the contents of one of the affected blocks

- Does this system have longrunning transactions? I'm basically wondering if
it is possible for the nightly VACFREEZE to have encountered the problematic
rows without having been able to remove them.

- Does the system use pg_repack/reorg or such? I've seen those cause all kinds
of corruption.

- What is the xid epoch? Best post all of pg_controldata.

- Have there been any crashes, failovers or such in the past? Sometimes older
corruption can manifest later (e.g. if you have an incorrectly initialized
PITR you can have "unconnected" segments beyond the current end of the
relation that "appear" once the relation grows sufficiently large)

- I'd check if you can figure out when the clog was truncated from the WAL,
that would provide a bound to which range of the WAL you need to be looking
at.

- Were there any graceful restarts of the system?

- What kernel version, distro and what kind of storage? As hinted above, it's
not unheard of for storage / filesystem bugs to lead to lost writes or lost
truncations (if a truncation fails / isn't permanent, it can have the effect
of basically making older data "reappear")

Greetings,

Andres Freund