Skip site navigation (1) Skip section navigation (2)

Help desperately needed: reoccurring invalid page header

From: Mauri Sahlberg <Mauri(dot)Sahlberg(at)claymountain(dot)com>
To: pgsql-admin(at)postgresql(dot)org
Cc: peikko(at)claymountain(dot)com, PoRRo <theporro(at)saunalahti(dot)fi>
Subject: Help desperately needed: reoccurring invalid page header
Date: 2005-04-28 06:27:30
Message-ID: 42708252.7070003@claymountain.com (view raw or flat)
Thread:
Lists: pgsql-admin
Hi,

Problem environment:
Database-server:
- Fedora Core 1 under virtuozzo virtual server running on 
Fujitsu-siemens hardware
- Kernel: Linux claymountain.planeetta.com 
2.4.20-021stab028.5.777-enterprise #1 SMP Tue Feb 22 17:44:46 MSK 2005 
i686 i686 i386 GNU/Linux
- Postgresql 7.4.6  (from rpm by provider: release: 2PGDG  Build Date: 
Mon 25 Oct 2004 11:25:56 PM EEST Build host gunduz.org)
- Query and update clients as php-applications
Database-update-client:
- Fedora Core 2
- Kernel Linux error.may.day 2.6.10-1.771_FC2 #1 Mon Mar 28 00:50:14 EST 
2005 i686 athlon i386 GNU/Linux
- A c++-client build against libpq++-4.0 from pgfoundry. I can not 
remember when the application was compiled but definitely not after auto 
upgrading client version to 7.4.7
- psql (PostgreSQL) 7.4.7

Problem description:
- After running out of money and several years of operation I moved 
database (7.4.6 from fedora RPM) from the client machine to the virtual 
server using pg_dump and pg_restore over the Internet and modified 
update client to contact remote host instead of localhost.
- After four months of operation on the virtual server the table 
"raakaleimat" - the one that the client mainly does inserts into - 
crashed with  "Invalid page header". After some googling I decided to 
drop the table, recreate it, restore data from previous backup as 
inserts and restart the database
- After four and half months of operation, a week ago, this happened 
again, and again I dropped the table, recreated it and restarted the 
database
- Now another table "firmat" that holds keys almost every other table in 
the database crashed with "Invalid page header". And what is worse  
another non application related database has failed with "Invalid page 
header".

It seems to me that the problem is  missing log-file 
"/var/lib/pgsql/data/pg_clog/0000" which I failed to spot on 21st of 
April when
the raakaleimat-table  crashed  second  time.

What can I do to recreate the missing file or is that what I should do? 
There are 13 databases on the system. Should I back them all up, drop 
everything. Remove every file, do initdb and restore everything? Should 
I kick the providers database rpm out of the box and fetch another one 
from somewhere else? Or should I downgrade client environment and 
recompile ?

Logs:
Apr 21 14:07:45 claymountain postgres[6453]: [8-1] FATAL:  the database 
system is starting up
Apr 21 14:07:45 claymountain postgres[5912]: [16-1] LOG:  database 
system is ready
Apr 21 14:08:35 claymountain postgres[7155]: [8-2] DETAIL:  could not 
open file
"/var/lib/pgsql/data/pg_clog/0295": No such file or directory
Apr 21 14:10:02 claymountain postgres[11079]: [9-2] DETAIL:  could not 
open file "/var/lib/pgsql/data/pg_clog/0000": No such file or directory
Apr 21 14:11:11 claymountain postgres[22476]: [8-1] LOG:  received fast 
shutdown request
Apr 21 14:11:11 claymountain postgres[22476]: [9-1] LOG:  aborting any 
active transactions
Apr 21 14:11:11 claymountain postgres[12042]: [10-1] FATAL:  terminating 
connection due to administrator command
Apr 21 14:11:11 claymountain postgres[12016]: [10-1] FATAL:  terminating 
connection due to administrator command
Apr 21 14:11:21 claymountain postgres[13074]: [11-1] LOG:  database 
system is shut down
Apr 21 14:11:42 claymountain postgres[13614]: [1-1] LOG:  could not 
create IPv6
socket: Address family not supported by protocol
Apr 21 14:11:42 claymountain postgres[13618]: [2-1] LOG:  database 
system was shut down at 2005-04-21 14:11:21 EEST
Apr 21 14:11:42 claymountain postgres[13618]: [3-1] LOG:  checkpoint 
record is at 6/350B0FDC
Apr 21 14:11:42 claymountain postgres[13618]: [4-1] LOG:  redo record is 
at 6/350B0FDC; undo record is at 0/0; shutdown TRUE
Apr 21 14:11:42 claymountain postgres[13618]: [5-1] LOG:  next 
transaction ID: 50072169; next OID: 3492415
Apr 21 14:11:42 claymountain postgres[13618]: [6-1] LOG:  database 
system is ready
Apr 21 14:13:38 claymountain postgres[15582]: [3-2] DETAIL:  could not 
open file "/var/lib/pgsql/data/pg_clog/0000": No such file or directory
...
Apr 28 08:08:03 claymountain postgres[3227]: [3-2] DETAIL:  could not 
open file
"/var/lib/pgsql/data/pg_clog/0000": No such file or directory
Apr 28 07:59:33 claymountain postgres[25623]: [2-1] ERROR:  invalid page 
header
in block 1 of relation "firmat"
This is on database pikkusisko, the one that has crashed twice before
Apr 28 08:12:40 claymountain postgres[10835]: [2-1] LOG:  recycled 
transaction log file "00000006000000B1"
Apr 28 08:41:05 claymountain postgres[7602]: [4-1] ERROR:  invalid page 
header in block 3 of relation "serendipity_suppress"
This one is one of the four serendipity databases we have, I do not yet 
know which one
Apr 28 08:46:45 claymountain postgres[14998]: [3-2] DETAIL:  could not 
open file "/var/lib/pgsql/data/pg_clog/0000": No such file or directory

Regards,
Mauri Sahlberg

Responses

pgsql-admin by date

Next:From: Ian WestmacottDate: 2005-04-28 12:58:18
Subject: Re: Help desperately needed: reoccurring invalid page
Previous:From: Alvaro HerreraDate: 2005-04-27 20:41:01
Subject: Re: catalog of postgres

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group