Re: [SPAM] Re: [SPAM] Re: BUG #3484: Missing pg_clog file / corrupt index

We've got the invalid page header again:

ERROR: invalid page header in block 18223 of relation "calllegstart"

bash-3.00# pg_controldata /var/lib/pgsql/data
pg_control version number: 812
Catalog version number: 200510211
Database system identifier: 5101280766299435989
Database cluster state: in production
pg_control last modified: Mon Aug 27 17:32:12 2007
Current log file ID: 6
Next log file segment: 242
Latest checkpoint location: 6/EFB4C658
Prior checkpoint location: 6/EDC84A9C
Latest checkpoint's REDO location: 6/EFB04720
Latest checkpoint's UNDO location: 0/0
Latest checkpoint's TimeLineID: 1
Latest checkpoint's NextXID: 23661755
Latest checkpoint's NextOID: 24576
Latest checkpoint's NextMultiXactId: 1
Latest checkpoint's NextMultiOffset: 0
Time of latest checkpoint: Mon Aug 27 17:29:17 2007
Maximum data alignment: 4
Database block size: 8192
Blocks per segment of large relation: 131072
Bytes per WAL segment: 16777216
Maximum length of identifiers: 64
Maximum columns in an index: 32
Date/time type storage: floating-point numbers
Maximum length of locale name: 128
LC_COLLATE: C
LC_CTYPE: C

-----Original Message-----
From: Feng Chen
Sent: Friday, August 24, 2007 1:42 PM
To: 'Tom Lane'; Stefan Kaltenbrunner
Cc: Alvaro Herrera; Gregory Stark; Marc Schablewski; Decibel!;
pgsql-bugs(at)postgresql(dot)org
Subject: RE: [SPAM] Re: [SPAM] Re: [BUGS] BUG #3484: Missing pg_clog
file / corrupt index

Yes, most of the time PostgreSQL only complains about missing pg_clog/
files then complains about invalid page headers. The "could not create
... File exists" message was only seen a couple of times.

I don't have a system that exhibits this problem right now. I can check
the XID counter when it happens again.

Thanks!

-----Original Message-----
From: Tom Lane [mailto:tgl(at)sss(dot)pgh(dot)pa(dot)us]
Sent: Friday, August 24, 2007 12:11 PM
To: Stefan Kaltenbrunner
Cc: Feng Chen; Alvaro Herrera; Gregory Stark; Marc Schablewski;
Decibel!; pgsql-bugs(at)postgresql(dot)org
Subject: [SPAM] Re: [SPAM] Re: [BUGS] BUG #3484: Missing pg_clog file /
corrupt index

Stefan Kaltenbrunner <stefan(at)kaltenbrunner(dot)cc> writes:
> Feng Chen wrote:
>> VERSION = PostgreSQL 8.1.2
> this could be
> http://archives.postgresql.org/pgsql-committers/2006-01/msg00287.php

Note, however, that that only explains the "could not create ... File
exists" complaint, which is hardly the main problem here.

The combination of "invalid page header" and "could not access status of
transaction" messages looks like nothing so much as severe data
corruption. Remember that the xmin/xmax fields are basically the first
thing we can check with any degree of strictness when examining a
tuple. This means that if a page is partially clobbered, but not in a
way that sets off the invalid-page-header checks, then the odds are very
high that the first detectable sign of trouble will be references to
transaction numbers that are far away from what the system is really
using. (Is 1481866610 anywhere near the current XID counter reported
by pg_controldata?)

I concur with the upthread comment to check into kernel bugs,
particularly if all of your machines are using the same old kernel
release.

regards, tom lane