Skip site navigation (1) Skip section navigation (2)

Re: [GENERAL] Undetected corruption of table files

From: "Albe Laurenz" <all(at)adv(dot)magwien(dot)gv(dot)at>
To: "Tom Lane *EXTERN*" <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: [GENERAL] Undetected corruption of table files
Date: 2007-08-28 08:14:17
Message-ID: D960CB61B694CF459DCFB4B0128514C2220A7F@exadv11.host.magwien.gv.at (view raw or flat)
Thread:
Lists: pgsql-generalpgsql-hackers
Tom Lane wrote:
>>>> Would it be an option to have a checksum somewhere in each
>>>> data block that is verified upon read?
> 
>>> That's been proposed before and rejected before.  See the 
>>> archives ...
> 
> I think
> the prior discussions were around the same time WAL was initially put
> in, and/or when we dropped the WAL CRC width from 64 to 32 bits.
> The very measurable overhead of WAL CRCs are the main thing that's
> discouraged us from having page CRCs.  (Well, that and the lack of
> evidence that they'd actually gain anything.)

Hmmm - silence me if I'm misunderstanding this, but the most
conclusive hit I had was a mail by you:

http://archives.postgresql.org/pgsql-general/2001-10/msg01142.php

which only got affirmative feedback.

Also, there's a TODO entry:

- Add optional CRC checksum to heap and index pages

This seems to me to be exactly what I wish for...

To the best of my knowledge, the most expensive thing in databases
today is disk I/O, because CPU speed is increasing faster. Although
calculating a checksum upon writing a block to disk will
certainly incur CPU overhead, what may have seemed too expensive
a couple of years ago could be acceptable today.

I understand the argument that it's the task of hardware and
OS to see that data don't get corrupted, but it would improve
PostgreSQL's reliabitity if it can detect such errors and at
least issue a warning.
This wouldn't fix the underlying problem, but it would tell you
to not overwrite last week's backup tape...

Not all databases are on enterprise scale storage systems, and
there's also the small possibility of PostgreSQL bugs that could
be detected that way.

Yours,
Laurenz Albe

In response to

Responses

pgsql-hackers by date

Next:From: EnricoDate: 2007-08-28 08:15:41
Subject: Diffondete......
Previous:From: Tom LaneDate: 2007-08-28 05:49:14
Subject: Re: Insufficient attention to security in contrib (mostly)

pgsql-general by date

Next:From: Dave PageDate: 2007-08-28 08:21:31
Subject: Re: Windows Download
Previous:From: Richard HuxtonDate: 2007-08-28 08:10:50
Subject: Re: delete vs insert vs update due to primary key dups -> which is better

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group