"PANIC: btree_split_redo: lost left sibling" problem

From: MStringham(at)tomax(dot)com
To: pgsql-admin(at)postgresql(dot)org
Subject: "PANIC: btree_split_redo: lost left sibling" problem
Date: 2008-12-04 18:27:37
Message-ID: OFF3E761DE.99B8E506-ON87257515.00647BD8-87257515.00656A7C@tomax.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-admin

This is my first post so hopefully I can provide all the information to
get this solved.
Last night we ran out of disk space on our server and when I tried to
start pg_ctl I got this error:
** EnterpriseDB Dynamic Tuning Agent ************************
* System Utilization: 66 % *
* Database Version: 8.1.3.12 *
* Operating System: Linux *
* RAM: 7.56 GB *
* Shared Memory: 7.55 GB *
* Max DB Connections: 100 *
*************************************************************

2008-12-04 10:20:47 MST LOG: database system was interrupted while in
recovery at 2008-12-04 10:11:35 MST
2008-12-04 10:20:47 MST HINT: This probably means that some data is
corrupted and you will have to use the last backup for recovery.
2008-12-04 10:20:47 MST LOG: starting archive recovery
2008-12-04 10:20:47 MST LOG: restore_command = "cp
/u01/home/enterprisedb/bkp/archivedir/%f %p"
cp: cannot stat `/u01/home/enterprisedb/bkp/archivedir/00000003.history':
No such file or directory
cp: cannot stat
`/u01/home/enterprisedb/bkp/archivedir/00000003000000350000002D': No such
file or directory
2008-12-04 10:20:47 MST LOG: checkpoint record is at 35/B74729F0
2008-12-04 10:20:47 MST LOG: redo record is at 35/B74729F0; undo record
is at 0/0; shutdown FALSE
2008-12-04 10:20:47 MST LOG: next transaction ID: 24775805; next OID:
730532
2008-12-04 10:20:47 MST LOG: next MultiXactId: 2952; next
MultiXactOffset: 6359
2008-12-04 10:20:47 MST LOG: automatic recovery in progress
2008-12-04 10:20:47 MST LOG: redo starts at 35/B7472A40
2008-12-04 10:20:47 MST PANIC: btree_split_redo: lost left sibling
2008-12-04 10:20:47 MST LOG: startup process (PID 2520) was terminated by
signal 6
2008-12-04 10:20:47 MST LOG: aborting startup due to startup process
failure

I went to the archivedir directory and indeed the files mentioned above
are not there. I did go to the pg_xlog folder and found that the files
listed above are there - this directory has a lot of data. I wanted to do
a PITR but to do this I need the database running. Here are the details:
EnterpriseDB version: 8.1.3.12
Server: Red Hat Enterprise Linux AS release 3 (Taroon Update 7)

I'm at a loss what to do. I've researched and couldn't find any answers.
I don't have a lot of space to work with still because I was going to
remove a bunch of old schemas that were no longer used but I cannot do
that until the db is up and running. Any ideas on what do do here?

Matt

CONFIDENTIALITY NOTICE: This message and any attachment(s) are solely for the use of the intended recipient(s) identified above and may contain information that is proprietary, privileged, or confidential. If you are not an intended recipient, you may not review, retransmit, or otherwise use this message or any attachment. If you have received this message in error, please immediately notify the sender by reply e-mail and delete this message.

Responses

Browse pgsql-admin by date

  From Date Subject
Next Message Marc Fromm 2008-12-04 19:18:32 Re: restore a table in a database
Previous Message Jaume Sabater 2008-12-04 17:29:11 Re: Using text search for locations, or a computed btree index