Skip site navigation (1) Skip section navigation (2)

Re: Block-level CRC checks

From: Gregory Stark <stark(at)enterprisedb(dot)com>
To: Paul Schlie <schlie(at)comcast(dot)net>
Cc: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, Brian Hurt <bhurt(at)janestcapital(dot)com>, <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Block-level CRC checks
Date: 2008-10-01 17:32:58
Message-ID: 87wsgsz0dx.fsf@oxford.xeocode.com (view raw or flat)
Thread:
Lists: pgsql-hackers
Paul Schlie <schlie(at)comcast(dot)net> writes:

> Tom Lane wrote:
>> Paul Schlie writes:
>>> - yes, if you're willing to compute true CRC's as opposed to simpler
>>> checksums, which may be worth the price if in fact many/most data
>>> check failures are truly caused by single bit errors somewhere in the
>>> chain,
>> 
>> FWIW, not one of the corrupted-data problems I've investigated has ever
>> looked like a single-bit error.  So the theoretical basis for using a
>> CRC here seems pretty weak.  I doubt we'd even consider automatic repair
>> attempts anyway.
>
> - although I accept that you may be correct in your assessment that most
> errors are in fact multi-bit; 

I've seen bad memory in a SCSI controller cause single-bit errors in storage.
It was quite confusing since the symptom was syntax errors in the C code we
were compiling on the server. The sysadmin actually caught it reliably
corrupting a block of source text written out and read back.

I've also seen single-bit errors caused by bad memory in a network interface.
*Twice*. Particularly nasty since the CRC on TCP/IP packets is only 16-bit so
a large enough ftp transfer would eventually finish despite the packet loss
but with the occasional bits flipped. In these days of SAN/NAS and SCSI over
IP that's pretty scary...

Several cases on list have come down to "filesystem secretly replaces entire
block of data with Folger's Crystals(tm) -- let's see if the database
notices". Any checksum would help in that case but I wouldn't discount single
bit errors either.

-- 
  Gregory Stark
  EnterpriseDB          http://www.enterprisedb.com
  Ask me about EnterpriseDB's PostGIS support!

In response to

Responses

pgsql-hackers by date

Next:From: Josh BerkusDate: 2008-10-01 17:47:07
Subject: September CommitFest Closed
Previous:From: Gregory StarkDate: 2008-10-01 17:25:52
Subject: Re: Block-level CRC checks

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group