Crash with pg_clog file not found

From: "Matthieu Roger" <matthieu(dot)roger(at)gene6(dot)com>
To: pgsql-admin(at)postgresql(dot)org
Subject: Crash with pg_clog file not found
Date: 2008-11-03 07:41:50
Message-ID: 30111ec50811022341q688ca946j8269cee93b206a3d@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-admin

Hello,

OS: Windows 2003 64bits
Cpu : 2x Opteron Dual core
Ram : 8Go Ram
Disk : Areca Raid10 6x 200Go sata
Postgresql 8.3.4

we're experiencing problems since some weeks with always the same
error (same transaction 0 and same pg_clog file 0000) :

2008-10-31 20:14:27 CET 10.0.0.119 exile PANIC: could not access
status of transaction 0
2008-10-31 20:14:27 CET 10.0.0.119 exile DETAIL: Could not open file
"pg_clog/0000": No such file or directory.
2008-10-31 20:14:27 CET 10.0.0.119 exile STATEMENT: SELECT
sp_execute_processes();

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

2008-10-31 20:17:32 CET 10.0.0.119 exile FATAL: connection limit
exceeded for non-superusers

[...]

We try to stop the service :

2008-10-31 20:39:30 CET LOG: received fast shutdown request
2008-10-31 20:39:30 CET LOG: aborting any active transactions
2008-10-31 20:39:30 CET 10.0.0.119 exile FATAL: terminating
connection due to administrator command

Line above repeated again and again

2008-10-31 20:44:20 CET LOG: server process (PID 2772) exited with
exit code 3
2008-10-31 20:44:20 CET LOG: terminating any other active server processes
2008-10-31 20:44:20 CET 10.0.0.119 vitevendu WARNING: terminating
connection because of crash of another server process
2008-10-31 20:44:20 CET 10.0.0.119 vitevendu DETAIL: The postmaster
has commanded this server process to roll back the current transaction
and exit, because another server process exited abnormally and
possibly corrupted shared memory.
2008-10-31 20:44:20 CET 10.0.0.119 vitevendu HINT: In a moment you
should be able to reconnect to the database and repeat your command.

Last block above repeated with all bases many times, then :

2008-10-31 20:44:21 CET 10.0.0.119 exile FATAL: the database system
is shutting down
[...]
2008-10-31 20:44:22 CET LOG: abnormal database system shutdown

Event log :
20:14:32 : Faulting application postgres.exe, version 8.3.4.8262,
faulting module postgres.exe, version 8.3.4.8262, fault address
0x0024a529.

When it crashes the service can't be restarted, no postmaster.pid file
to remove, it does not want to restart, we need to reboot server then
it recovers but we have strange data and duplicated content so we need
to recover from a backup.

We vacuum regularly, autovacuum is enabled, cpu usage is ok (<50%) as
well as mem usage. I set log to debug1.

I don't know what the problem can be :-/

New crash with debug1 (not much info) :

2008-11-03 01:15:45 CET 10.0.0.119 exile LOG: 00000: duration:
2688.000 ms statement: SELECT sp_execute_processes();
2008-11-03 01:15:45 CET 10.0.0.119 exile LOCATION: exec_simple_query,
.\src\backend\tcop\postgres.c:1063
2008-11-03 01:15:46 CET 10.0.0.119 exile PANIC: 58P01: could not
access status of transaction 0
2008-11-03 01:15:46 CET 10.0.0.119 exile DETAIL: Could not open file
"pg_clog/0000": No such file or directory.
2008-11-03 01:15:46 CET 10.0.0.119 exile LOCATION: SlruReportIOError,
.\src\backend\access\transam\slru.c:845
2008-11-03 01:15:46 CET 10.0.0.119 exile STATEMENT: SELECT
sp_execute_processes();

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

Matthieu

Responses

Browse pgsql-admin by date

  From Date Subject
Next Message Rainer Bauer 2008-11-03 11:01:00 Re: Crash with pg_clog file not found
Previous Message Scott Marlowe 2008-11-02 23:35:04 Re: pg_dump's table is empty