Re: 64-bit MultiXactOffset vs. 9.3->9.4 upgrade, pg_resetwal, "wraparound" msg

On 16/09/2026 14:33, Heikki Linnakangas wrote:
> On 28/08/2026 02:17, Noah Misch wrote:
>> commit bd8d9c9 wrote:
>>> Commit:     Heikki Linnakangas <heikki(dot)linnakangas(at)iki(dot)fi>
>>> CommitDate: Tue Dec 9 13:53:03 2025 +0200
>>>
>>>      Widen MultiXactOffset to 64 bits
>>
>>> --- a/src/bin/pg_upgrade/pg_upgrade.c
>>> +++ b/src/bin/pg_upgrade/pg_upgrade.c
>>
>>> +        nxtmulti = old_cluster.controldata.chkpnt_nxtmulti;
>>> +        if (old_cluster.controldata.cat_ver >=
>>> MULTIXACT_FORMATCHANGE_CAT_VER)
>>> +        {
>>> +            /* Versions 9.3 - 18: convert all multixids  */
>>> +            oldstMulti = old_cluster.controldata.chkpnt_oldstMulti;
>>
>> If a cluster's upgrade history includes an upgrade from 9.3 to early
>> 9.4, it
>> may have a wrong value here.  Specifically, upgrades done before a61daa14
>> (2014-07 commit) have that hazard.  We still have backend code to
>> detect such
>> cases and reduce damage:
>>
>>         ereport(LOG,
>>                 (errmsg("cannot truncate up to MultiXact %u because it
>> does not exist on disk, skipping truncation",
>>                         newOldestMulti)));
>>
>> However, the pg_upgrade side from the v19 commit lacks such
>> protection.  If
>> heap tuples still reference older multixacts than the faulty control data
>> suggests, pg_upgrade will copy too small a range, making affected tuples
>> unreadable.
>
> Thanks, I'll look into this. My first reaction is that I think if
> oldstMulti is incorrectly too old, the upgrade will fail because the
> conversion routine will fail to find it. If it's too new, i.e. "in the
> future", it will also fail to find it.
>
> The third possibility is that the bogus oldstMulti value is within the
> range of the "real" range. That can happen if multixid wraparound had
> already happened before the (broken) 9.3 -> 9.4 upgrade. In that case,
> even if the multixids are still readable on disk, you're one vacuum away
> from truncating them. In other words, the damage has already been done,
> or could be done at any minute. That's the scenario for which the commit
> message 78db307bb2 says "this mechanism cannot save us".
>
> But I'll do some testing of that. Let's ensure that the error message
> makes sense at the very least.

Ok, I reconstructed that scenario artificially with the attached script.
It creates a new cluster, runs a test program to create a bunch of
multixids, runs vacuum freeze to truncate the multixids SLRU and advance
oldestMulti. Finally, it uses pg_resetwal to reset oldestMulti back to
1, to simulate what you'd get with the buggy 9.3 pg_upgrade. I tested
this with a v18 source cluster, but I don't think the source version
matters. (It must've been pg_upgraded to 9.3 in the past, but it
could've gone through more pg_upgrades since)

When you run pg_upgrade on that, it fails like this:

> Converting pg_multixact files
> could not open file "/tmp/pgcluster/pg_multixact/offsets/0000": No such file or directory

Not an ideal error message, but at least you get an error rather than
silently losing data.

With the attached patch, you get an error message crafted specifically
for that case.

> Converting pg_multixact files fatal
>
> Segment file containing multixid 1 does not exist.
> This can happen if an old version of pg_upgrade was used in the past
> to upgrade the cluster from version 9.3 to 9.4. If that is the cause,
> it can be fixed by running VACUUM FREEZE before upgrade.
> Failure, exiting

I'm not sure this is worth committing though. How many clusters are
there out there that have been pg_upgraded to version 9.3.0-9.3.4, and
haven't since been running for long enough to fix the oldestMultiXid
value? Having that error message seems pretty harmless too, but if you
actually hit that case that the file is missing, it might actually be
more likely that you've hit some other bug than that old pg_upgrade issue.

Another approach would be to ignore the oldestMulti in the control file
and scan pg_multixact/offsets for the oldest segment that actually
exists. But I don't like that. Unless you actually hit that old
pg_upgrade bug, a missing file is a sign of trouble, and it's better
that you get alerted about it than upgrading the broken cluster to a new
version.

- Heikki