Re: Protecting against unexpected zero-pages: proposal

On Tue, Nov 09, 2010 at 02:05:57PM -0500, Robert Haas wrote:
> On Tue, Nov 9, 2010 at 12:31 PM, Greg Stark <gsstark(at)mit(dot)edu> wrote:
> > On Tue, Nov 9, 2010 at 5:06 PM, Aidan Van Dyk <aidan(at)highrise(dot)ca> wrote:
> >> So, for getting checksums, we have to offer up a few things:
> >> 1) zero-copy writes, we need to buffer the write to get a consistent
> >> checksum (or lock the buffer tight)
> >> 2) saving hint-bits on an otherwise unchanged page. ?We either need to
> >> just not write that page, and loose the work the hint-bits did, or do
> >> a full-page WAL of it, so the torn-page checksum is fixed
> >
> > Actually the consensus the last go-around on this topic was to
> > segregate the hint bits into a single area of the page and skip them
> > in the checksum. That way we don't have to do any of the above. It's
> > just that that's a lot of work.
>
> And it still allows silent data corruption, because bogusly clearing a
> hint bit is, at the moment, harmless, but bogusly setting one is not.
> I really have to wonder how other products handle this. PostgreSQL
> isn't the only database product that uses MVCC - not by a long shot -
> and the problem of detecting whether an XID is visible to the current
> snapshot can't be ours alone. So what do other people do about this?
> They either don't cache the information about whether the XID is
> committed in-page (in which case, are they just slower or do they have
> some other means of avoiding the performance hit?) or they cache it in
> the page (in which case, they either WAL log it or they don't checksum
> it). I mean, there aren't any other options, are there?
>
> --
> Robert Haas
> EnterpriseDB: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>

That would imply that we need to have a CRC for just the hint bit
section or some type of ECC calculation that can detect bad hint
bits independent of the CRC for the rest of the page.

Regards,
Ken