Re: Online verification of checksums

On 11/30/20 9:27 AM, Stephen Frost wrote:
> Greetings,
>
> * Michael Paquier (michael(at)paquier(dot)xyz) wrote:
>> On Fri, Nov 27, 2020 at 11:15:27AM -0500, Stephen Frost wrote:
>>> * Magnus Hagander (magnus(at)hagander(dot)net) wrote:
>>>> On Thu, Nov 26, 2020 at 8:42 AM Michael Paquier <michael(at)paquier(dot)xyz> wrote:
>>>>> But here the checksum is broken, so while the offset is something we
>>>>> can rely on how do you make sure that the LSN is fine? A broken
>>>>> checksum could perfectly mean that the LSN is actually *not* fine if
>>>>> the page header got corrupted.
>>>
>>> Of course that could be the case, but it gets to be a smaller and
>>> smaller chance by checking that the LSN read falls within reasonable
>>> bounds.
>>
>> FWIW, I find that scary.
>
> There's ultimately different levels of 'scary' and the risk here that
> something is actually wrong following these checks strikes me as being
> on the same order as random bits being flipped in the page and still
> getting a valid checksum (which is entirely possible with our current
> checksum...), or maybe even less.

I would say a lot less. First you'd need to corrupt one of the eight
bytes that make up the LSN (pretty likely since corruption will probably
affect the entire block) and then it would need to be updated to a value
that falls within the current backup range, a 1 in 16 million chance if
a terabyte of WAL is generated during the backup. Plus, the corruption
needs to happen during the backup since we are going to check for that,
and the corrupted LSN needs to be ascending, and the LSN originally read
needs to be within the backup range (another 1 in 16 million chance)
since pages written before the start backup checkpoint should not be torn.

So as far as I can see there are more likely to be false negatives from
the checksum itself.

It would also be easy to add a few rounds of checks, i.e. test if the
LSN ascends but stays in the backup LSN range N times.

Honestly, I'm much more worried about corruption zeroing the entire
page. I don't know how likely that is, but I know none of our proposed
solutions would catch it.

Andres, since you brought this issue up originally perhaps you'd like to
weigh in?

Regards,
--
-David
david(at)pgmasters(dot)net