Re: 9.0.4 Data corruption issue

Ken Caruso <ken(at)ipl31(dot)net> writes:
> Sorry, the actual error reported by CLUSTER is:

> gpup=> cluster verbose tablename;
> INFO: clustering "dbname.tablename"
> WARNING: could not write block 12125253 of base/2651908/652397108
> DETAIL: Multiple failures --- write error might be permanent.
> ERROR: could not open file "base/2651908/652397108.1" (target block
> 12125253): No such file or directory
> CONTEXT: writing block 12125253 of relation base/2651908/652397108

Hmm ... it looks like you've got a dirty buffer in shared memory that
corresponds to a block that no longer exists on disk; in fact, the whole
table segment it belonged to is gone. Or maybe the block or file number
in the shared buffer header is corrupted somehow.

I imagine you're seeing errors like this during each checkpoint attempt?

I can't think of any very good way to clean that up. What I'd try here
is a forced database shutdown (immediate-mode stop) and see if it starts
up cleanly. It might be that whatever caused this has also corrupted
the back WAL and so WAL replay will result in the same or similar error.
In that case you'll be forced to do a pg_resetxlog to get the DB to come
up again. If so, a dump and reload and some manual consistency checking
would be indicated :-(

regards, tom lane