invalid OID warning after disk failure

From: Gabriel Sánchez Martínez <gabrielesanchez(at)gmail(dot)com>
To: Postgres General <pgsql-general(at)postgresql(dot)org>
Subject: invalid OID warning after disk failure
Date: 2014-11-17 04:50:28
Message-ID: 54697E94.6050007@gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

I'm running PostgreSQL 9.3.5 on Ubuntu 14.04 on x86_64. The database
directory is on linux mdadm RAID10, using 4 4TB disks and a far=2
layout. While the RAID tolerates 1 drive failure nicely, I had the
misfortune of 2 drives failing consecutively, one of which had many
sectors reallocated and began failing SMART criteria. That one is out
now. As a result of this some files were corrupted.

I was getting the following errors on some tables:

ERROR: could not read block 0 in file "base/27810/3995569":
Input/output error

but after dropping those tables the errors are gone.

The situation appears to be stable now, but upon running REINDEX and
VACUUM on one of the databases, I get the following:

WARNING: relation "pg_attrdef" TID 1/1: OID is invalid
WARNING: relation "pg_attrdef" TID 1/2: OID is invalid
WARNING: relation "pg_attrdef" TID 1/3: OID is invalid
...

Should I drop the database and restore it from a backup? My most recent
backup is from late September, so I would lose some data. I also backed
up what I could as soon as the disks started giving errors, but I don't
know if I can trust that.

Should I drop the entire cluster?

Regarding hardware, I'm going to add hot standby drives to prevent this
from happening in the future.

Thanks in advance for your advice.

Regards,
Gabriel

Responses

Browse pgsql-general by date

  From Date Subject
Next Message Raghu Ram 2014-11-17 08:28:12 Re: invalid OID warning after disk failure
Previous Message Steve Atkins 2014-11-17 01:26:30 Re: Comparing results of regexp_matches