Re: Hash index corruption and CLOG lookup errors from checkpoint races

From: vignesh C <vignesh21(at)gmail(dot)com>
To: shihao zhong <zhong950419(at)gmail(dot)com>
Cc: pgsql-hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org>, Robert Haas <robertmhaas(at)gmail(dot)com>, Heikki Linnakangas <hlinnaka(at)iki(dot)fi>, Amit Kapila <amit(dot)kapila16(at)gmail(dot)com>
Subject: Re: Hash index corruption and CLOG lookup errors from checkpoint races
Date: 2026-11-02 12:26:07
Message-ID: CALDaNm1+SiZ9SGt=X9KpOAx4vG8ZjrsXSq1o4nEiP7C_=NDGnA@mail.gmail.com
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On Sun, 1 Nov 2026 at 06:35, shihao zhong <zhong950419(at)gmail(dot)com> wrote:
>
> Subject: Index corruption: two more checkpoint races
>
> Hi,
>
> Two more races like [1] and [2]. A checkpoint can start between a WAL
> record and the file change.
>
> 1. vac_truncate_clog() truncates CLOG before it advances oldestXid.
> After a crash, pg_xact_status() on a removed XID fails with "could not
> open file pg_xact/0000" instead of returning NULL. A client that
> checks an old transaction after a reconnect gets an error.
>
> The next vacuum does not fix it. It stays until another CLOG segment
> is removed.
>

Good catch.
The CLOG fix made me look at the other limits recorded in a checkpoint
and restored during crash recovery: oldestXid, oldestCommitTsXid and
oldestMulti. oldestCommitTsXid is already advanced before
TruncateCommitTs(), but oldestMulti has the same race that oldestXid
had. I was able to reproduce it.

Having a similar test for oldestMultiXact results in an oldest
checkpointed MultiXactId refers to a segment that no longer exists.
After a crash, this produces:
LOG: MultiXact member truncation is disabled because oldest checkpointed
MultiXact 1 does not exist on disk

Attached a one-line injection point in TruncateMultiXact(), after
WriteMTruncateXlogRec() and before the in-memory update (loaded before
START_CRIT_SECTION, since it
is hit inside it), and a TAP test for src/test/recovery
Thoughts?

Regards,
Vignesh

Attachment Content-Type Size
062_multixact_truncate_checkpoint.pl text/x-perl-script 6.9 KB
multixact-injection-point.diff application/octet-stream 1.1 KB

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Lucas Jeffrey 2026-11-02 12:38:58 Re: Re: [PATCH] Fix segmentation fault caused by reentrancy in RI_Fkey_cascade_del (ri_triggers.c)
Previous Message Sami Imseih 2026-11-02 12:11:29 Re: Report buffer access strategies in EXPLAIN