Re: Changing the state of data checksums in a running cluster

On 8/27/25 13:00, Daniel Gustafsson wrote:
>> On 27 Aug 2025, at 11:39, Tomas Vondra <tomas(at)vondra(dot)me> wrote:
>
>> Just to be clear - I don't see any pg_checksums failures either. I only
>> see failures in the standby log, and I don't think the script checks
>> that (it probably should).
>
> Right, that's what I'm been checking too. I have been considering adding
> another background process for monitoring all the log entries but I just
> thought of a much simpler solution. When the clusters are turned off we can
> take the opportunity to slurp the log written since last restart and inspect
> it. The attached adds this.
>

There's still a couple issues, unfortunately. First, this may not do
what you intended:

sub cointoss
{
return int(rand(2) == 1);
}

The rand() call returns values in [0, 2.0), it's very unlikely to
produce 1, and so there are no restarts. I fixed this to

sub cointoss
{
return int(rand() < 0.5);
}

and then it starts working, restarting with ~50% probability.

Then it hits another problem when calling pg_checksums. That only works
after a "clean" shutdown, not after immediate one.This can't just check
is_alive, it needs to remember how exactly was the server stopped and
only verify checksums for 'fast' mode. I never saw any failures in
pg_checksums, so I just commented that out.

Then it starts working as intended, I think. I only did a couple runs so
far, but I haven't found any checksum failures ... Then I realized I did
the earlier runs with slightly stale master HEAD (38c5fbd97ee6a to be
precise), while this time I did git pull.

And this happened on Friday:

commit c13070a27b63d9ce4850d88a63bf889a6fde26f0
Author: Alexander Korotkov <akorotkov(at)postgresql(dot)org>
Date: Fri Aug 22 18:44:39 2025 +0300

Revert "Get rid of WALBufMappingLock"

This reverts commit bc22dc0e0ddc2dcb6043a732415019cc6b6bf683.
It appears that conditional variables are not suitable for use
inside critical sections. If WaitLatch()/WaitEventSetWaitBlock()
face postmaster death, they exit, releasing all locks instead of
PANIC. In certain situations, this leads to data corruption.

...

I think it's very likely the checksums were broken by this. After all,
that linked thread has subject "VM corruption on standby" and I've only
ever seen checksum failures on standby on the _vm fork.

There's also an issue when calling teardown_node at the end. That won't
work if the instance got stopped in the last round, and the test will
fail like this:

t/006_concurrent_pgbench.pl .. 379/? # Tests were run but no plan was
declared and done_testing() was not seen.
# Looks like your test exited with 29 just after 379.
t/006_concurrent_pgbench.pl .. Dubious, test returned 29 (wstat 7424,
0x1d00)
All 379 subtests passed

> It would probably be good to at some point clean this up a little by placing
> all of variables for a single node in an associative hash which can be passed
> around, and place repeated code in subroutines etc..
>

Yeah. For me Perl is hard to read in any case, but if we can cleanup
this a little bit before additional cases, that'd be nice.

regards

--
Tomas Vondra