Re: Changing the state of data checksums in a running cluster

Hi,

On 2026-04-05 00:27:00 +0200, Daniel Gustafsson wrote:
> > On 4 Apr 2026, at 02:35, Daniel Gustafsson <daniel(at)yesql(dot)se> wrote:
> >
> >> On 4 Apr 2026, at 00:59, Daniel Gustafsson <daniel(at)yesql(dot)se> wrote:
> >>
> >>> On 3 Apr 2026, at 23:46, Daniel Gustafsson <daniel(at)yesql(dot)se> wrote:
> >>>
> >>> After many more runs on CI I ended up pushing this version, and I see BF
> >>> members being angry due the test not waiting for the launcher to exit. I am
> >>> working on a fix right now.
> >>
> >> 0036232ba8f seems to have made the failing animals slightly happier, I will
> >> continue to monitor the buildfarm for other fallout.
> >
> > The intermittent failure on kestrel implies timing similar to the one fixed in
> > 0036232ba8fb28, a tentative fix is to make it part of waiting for an endstate
> > (on or off) to make sure the cluster is always in the right state for new
> > operations. Right now kestrel is the one which has been flapping, I'm waiting
> > a bit to see if more will follow and give further clues.
>
> mylodon had the same failure, and I believe the bug is in my injection point
> test code. I have a tentative fix in the attached refactoring which moves over
> to using the injection_point extension module. It's still fairly rare so I'm
> holding off for a little bit before pushing it to see if I can collect a little
> bit more evidence.

There are a lot checksum related errors on CI:

https://cirrus-ci.com/task/4848298592305152
https://cirrus-ci.com/task/5338691381493760
https://cirrus-ci.com/task/6271077241847808
https://cirrus-ci.com/task/6150048418889728

They probably are mostly the issues you know about. It'd be nice to get them
fixed soon-ish...

Greetings,

Andres Freund