Skip site navigation (1) Skip section navigation (2)

Re: Block-level CRC checks

From: Greg Smith <gsmith(at)gregsmith(dot)com>
To: Heikki Linnakangas <heikki(dot)linnakangas(at)enterprisedb(dot)com>
Cc: Alvaro Herrera <alvherre(at)commandprompt(dot)com>, Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Block-level CRC checks
Date: 2008-09-30 19:23:28
Message-ID: Pine.GSO.4.64.0809301505050.18069@westnet.com (view raw or flat)
Thread:
Lists: pgsql-hackers
On Tue, 30 Sep 2008, Heikki Linnakangas wrote:

> Doesn't some filesystems include a per-block CRC, which would achieve the 
> same thing? ZFS?

Yes, there is a popular advoacy piece for ZFS with a high-level view of 
why and how they implement that at 
http://blogs.sun.com/bonwick/entry/zfs_end_to_end_data The guarantees are 
stronger than what you can get if you just put a CRC in the block itself. 
I'd never really thought too hard about putting this in the database 
knowing that ZFS is available for environments where this is a concern, 
but it certainly would be a nice addition.

The best analysis I've ever seen that makes a case for OS or higher level 
disk checksums of some sort, by looking at the myriad ways that disks and 
disk arrays fail in the real world, is in 
http://www.usenix.org/event/fast08/tech/full_papers/bairavasundaram/bairavasundaram.pdf 
(there is a shorter version that hits the high points of that at 
http://www.usenix.org/publications/login/2008-06/openpdfs/bairavasundaram.pdf 
)

One really interesting bit in there I'd never seen before is that they 
find real data that supports the stand that enterprise drives are 
significantly more reliable than consumer ones.  While general failure 
rates aren't that different, "SATA disks have an order of magnitude higher 
probability of developing checksum mismatches than Fibre Channel disks. We 
find that 0.66% of SATA disks develop at least one mismatch during the 
first 17 months in the field, whereas only 0.06% of Fibre Channel disks 
develop a mismatch during that time."

--
* Greg Smith gsmith(at)gregsmith(dot)com http://www.gregsmith.com Baltimore, MD

In response to

pgsql-hackers by date

Next:From: Dimitri FontaineDate: 2008-09-30 19:39:10
Subject: Re: Common Table Expressions (WITH RECURSIVE) patch
Previous:From: pgsqlDate: 2008-09-30 19:17:18
Subject: Re: Block-level CRC checks

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group