Offline data checksum changes can cause incorrect checksum state on standbys

Hi hackers,

while working on [1], I hit 2 issues involving offline checksum changes.

First one, is a case where a standby can enable checksum verification without its
own pages having been checksummed, making the standby unreadable.

The issue is due to f19c0eccae96 as the standby can enable checksum verification
from primary WAL.

Repro 1:

1/ create a primary and a standby both with checksum set to false
2/ stop the standby and the primary
3/ enable checksums only on the primary
4/ restart the primary and run checkpoint: this checkpoint XLOG_CHECKPOINT_REDO
record carries data_checksum_version=on. It gives the standby a WAL record that
makes it enable checksum verification.
5/ restart the standby: that will replay the primary’s new checksum state, despite
never having its own pages checksummed.
6/ try to connect to the standby: FATAL: invalid page in block 0 of relation "global/1260"

The second issue occurs when combining online and offline checksum transitions.

Repro 2:

1/ create a primary and a standby both with checksum set to false
2/ enable checksums online on the primary and wait that data_checksums is on
on the primary and on the standby
3/ stop only the standby
4/ disable checksums offline only on the standby
5/ restart the standby and check data_checksums. You'll see that it's still on
despites that we disabled it in step 4/

I initially considered detecting the mismatch during WAL replay and reporting an
error. Although this produces a clear error message, it does not help much in
practice because the standby cannot recover and has to be recreated.

Therefore, I think a simpler fix is to preserve the pre-f19c0eccae96 behavior
for offline checksum changes: they are not propagated through WAL. In the second
repro, the standby therefore remains off, honoring its local offline change.

This is what the attached patch proposes: it marks offline changes as local and
tracks the latest WAL-logged transition, so recovery ignores remote offline
states while still applying newer online transitions.

If this looks like too much code changes so close to the v19 release, another
option could be to remove pg_checksums --enable and --disable while keeping --check
and require checksum state changes to be done online.

As this is a 19 regression, I think it should be added as an open item.

Thoughts?

[1]: https://postgr.es/m/ajAAwSFy0WVMroyk%40bdtpg

Regards,

--
Bertrand Drouvot
PostgreSQL Contributors Team
RDS Open Source Databases
Amazon Web Services: https://aws.amazon.com