Re: Page Checksums

On 12/19/2011 02:44 PM, Kevin Grittner wrote:
> I was thinking that we would warn when such was found, set hint bits
> as needed, and rewrite with the new CRC. In the unlikely event that
> it was a torn hint-bit-only page update, it would be a warning about
> something which is a benign side-effect of the OS or hardware crash.
> The argument was that it could happen months later, and people
> might not remember the crash. My response to that is: don't let it
> wait that long. By forcing a vacuum of all possibly-affected tables
> (or all tables if the there's no way to rule any of them out), you
> keep it within recent memory.

Cleanup that requires a potentially unbounded in size VACUUM to sort out
doesn't sound like a great path to wander down. Ultimately any CRC
implementation is going to want a "scrubbing" feature like those found
in RAID arrays eventually, one that wanders through all database pages
looking for literal bitrot. And pushing in priority requests for things
to check to the top of its queue may end up being a useful feature
there. But if you need all that infrastructure just to get the feature
launched, that's a bit hard to stomach.

Also, as someone who follows Murphy's Law as my chosen religion, I would
expect this situation could be exactly how flaky hardware would first
manifest itself: server crash and a bad CRC on the last thing written
out. And in that case, the last thing you want to do is assume things
are fine, then kick off a VACUUM that might overwrite more good data
with bad. The sort of bizarre, "that should never happen" cases are the
ones I'd most like to see more protection against, rather than excusing
them and going on anyway.

> There's also *always" a possibility that CRC error is a false
> positive -- if only the bytes in the CRC were damaged. We're
> talking quantitative changes here, not qualitative.

The main way I expect to validate this sort of thing is with an as yet
unwritten function to grab information about a data block from a standby
server for this purpose, something like this:

Master: Computed CRC A, Stored CRC B; error raised because A!=B
Standby: Computed CRC C, Stored CRC D

If C==D && A==C, the corruption is probably overwritten bits of the CRC B.

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us