Re: Block-level CRC checks

Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:

> "Jonah H. Harris" <jonah(dot)harris(at)gmail(dot)com> writes:
>> On Wed, Oct 1, 2008 at 10:27 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>> Your optimism is showing ;-). XLogInsert routinely shows up as a major
>>> CPU hog in any update-intensive test, and AFAICT that's mostly from the
>>> CRC calculation for WAL records.
>
>> I probably wouldn't compare checksumming *every* WAL record to a
>> single block-level checksum.
>
> No, not at all. Block-level checksums would be an order of magnitude
> more expensive: they're on bigger chunks of data and they'd be done more
> often.

Yeah, it's not a single block, it's the total amount that matters and that's
going to amount to the entire i/o bandwidth of the database. That said I think
the reason WAL checksums are so expensive is the startup and finishing cost.

I wonder if we could do something clever here though. Only one process is busy
calculating the checksum -- it just has to know if anyone fiddles the hint
bits while it's busy.

If setting a hint bit cleared a flag on the buffer header then the
checksumming process could set that flag, begin checksumming, and check that
the flag is still set when he's finished.

Actually I suppose that wouldn't actually be good enough. He would have to do
the i/o and check that the checksum was still valid after the i/o. If not then
he would have to recalculate the checksum and repeat the i/o. That might make
the idea a loser since I think the only way it wins is if you rarely actually
get someone setting the hint bits during i/o anyways.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Get trained by Bruce Momjian - ask me about EnterpriseDB's PostgreSQL training!