Re: Offline data checksum changes can cause incorrect checksum state on standbys

Hi,

On Fri, Aug 28, 2026 at 04:16:55AM -0700, Zsolt Parragi wrote:
> Thanks!
>
> I agree that v4 is not a complete fix, and we could make it better.
> The question is the balance, as every change also makes it more
> complex. The main point of it is to try to minimize how invasive of a
> patch it is, and making sure that offline changes work and don't
> result in completely breaking a standby.

>
> 1: this is a valid issue, but requires somebody doing an offline
> change immediately after an online change.

I don't think it has to be immediate. The window lasts until the standby records
a restartpoint after the XLOG2_CHECKSUMS(on) record. That may happen considerably
later, depending on checkpoint replay and restartpoint creation.

> The effect is that in this
> print out a warning about this into the log, so this is visible, and
> everything will continue working.

Yes, but that leaves mismatched states and that's what we try to avoid.

> 2: my understanding is that if there's a concurrent checkpoint and a
> crash shortly after it, we might throw away an otherwise completed
> online checksum after restarting. We properly log that checksums were
> interrupted, and the state remains "off" on all nodes.

Right, but the online transition had reached on before the crash and is rolled
back because of 0001’s handling of the stale inprogress-on value. Even if all
nodes return to off, that still looks like an incorrect state rollback introduced
by 0001.

> While this is
> not ideal, I think this is an unlikely scenario

yeah, probably.

> and not the only such
> issue, for example a failing DROP DATABASE foo FORCE similarly can
> interrupt checksums in an unlikely case, as I reported in another
> thread.

I think our case is different.
The checksum state has reached on, and both XLOG2_CHECKSUMS and the later REDO
record carry on. It returns to off due to 0001.
The sleeps in the repro only make the possible interleaving deterministic.

> 3: also valid, but in my repro of this the warning fired, so it's not
> silent, and things seem to work fine after the warning, and the user
> can issue either an online or an offline change.

The warning is useful, but I don't think it makes the resulting state correct.
I think that leaves precisely the mismatched state we are trying to avoid.

> 4: I couldn't construct a repro for this case, I think this can only
> happen in theory in very specific engineered scenarios

Yes, it's doable. While that does not lead to correctness issue, it still
questions the logic here.

> I agree that combining the two patches would be the best solution in
> the warning direction, e.g. solving 1+3 requires the pg_control
> changes from v1.

Yeah, but if the resulting patch ends up being significantly more complex, that
would not be reassuring either.

> The reason I left that out is what I started with in
> this reply: simplicity. I was mainly considering combining the two
> because of v4 can emit spurious warnings in some cases

Does that refer to the cases I reported for v4, or did you have additional cases
in mind?

> additional log state stating the correction), but even with these I am
> not sure if we should make it more complex, as none of these result in
> crashes/data corruption, only in state rolling back in some
> engineering situations.

Yeah, I understand each of these tradeoffs in isolation. What concerns me is their
cumulative effect: we moved from wanting to reject mismatched states to warning
about them, and we are now considering leaving some known rollback cases unhandled
to keep the patch manageable.

Also, I’m not sure 1 and 3 are limited to engineered situations. Even without
immediate crashes, they can leave the nodes with mismatched states, which is
what the patchset is trying to address.

> I'll try to look into what adding the two
> patches together looks like, but it most likely combines their size,
> as they improve the current master code in different ways.

Thanks!

Regards,

--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com