Skip site navigation (1) Skip section navigation (2)

Re: Block-level CRC checks

From: Gregory Stark <stark(at)enterprisedb(dot)com>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: "Jonah H(dot) Harris" <jonah(dot)harris(at)gmail(dot)com>, pgsql(at)mohawksoft(dot)com, "Hannu Krosing" <hannu(at)2ndquadrant(dot)com>, Decibel! <decibel(at)decibel(dot)org>, "Alvaro Herrera" <alvherre(at)commandprompt(dot)com>, "Pg Hackers" <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Block-level CRC checks
Date: 2008-10-01 15:57:50
Message-ID: 87fxng1f5t.fsf@oxford.xeocode.com (view raw or flat)
Thread:
Lists: pgsql-hackers
Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:

> "Jonah H. Harris" <jonah(dot)harris(at)gmail(dot)com> writes:
>> On Wed, Oct 1, 2008 at 10:27 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>> Your optimism is showing ;-).  XLogInsert routinely shows up as a major
>>> CPU hog in any update-intensive test, and AFAICT that's mostly from the
>>> CRC calculation for WAL records.
>
>> I probably wouldn't compare checksumming *every* WAL record to a
>> single block-level checksum.
>
> No, not at all.  Block-level checksums would be an order of magnitude
> more expensive: they're on bigger chunks of data and they'd be done more
> often.

Yeah, it's not a single block, it's the total amount that matters and that's
going to amount to the entire i/o bandwidth of the database. That said I think
the reason WAL checksums are so expensive is the startup and finishing cost.

I wonder if we could do something clever here though. Only one process is busy
calculating the checksum -- it just has to know if anyone fiddles the hint
bits while it's busy.

If setting a hint bit cleared a flag on the buffer header then the
checksumming process could set that flag, begin checksumming, and check that
the flag is still set when he's finished.

Actually I suppose that wouldn't actually be good enough. He would have to do
the i/o and check that the checksum was still valid after the i/o. If not then
he would have to recalculate the checksum and repeat the i/o. That might make
the idea a loser since I think the only way it wins is if you rarely actually
get someone setting the hint bits during i/o anyways.

-- 
  Gregory Stark
  EnterpriseDB          http://www.enterprisedb.com
  Get trained by Bruce Momjian - ask me about EnterpriseDB's PostgreSQL training!

In response to

Responses

pgsql-hackers by date

Next:From: Jonah H. HarrisDate: 2008-10-01 15:58:42
Subject: Re: Block-level CRC checks
Previous:From: Alvaro HerreraDate: 2008-10-01 15:57:31
Subject: Re: Block-level CRC checks

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group