Re: Postgres database is not logging any messages after the reboot

Here are the messages from the pg_log before and after server reboot.

2007-10-10 06:22:50 CDT LOG: archived transaction log file
"000000010000007000000090"
2007-10-10 06:29:04 CDT LOG: autovacuum: processing database "nsc"
2007-10-10 06:34:58 CDT LOG: archived transaction log file
"000000010000007000000091"
2007-10-10 07:11:48 CDT LOG: database system was interrupted at
2007-10-10 06:40:05 CDT
2007-10-10 07:11:48 CDT LOG: checkpoint record is at 70/925E7B68
2007-10-10 07:11:48 CDT LOG: redo record is at 70/925E7B68; undo
record is at 0/0; shutdown FALSE
2007-10-10 07:11:48 CDT LOG: next transaction ID: 656073787; next
OID: 2915367
2007-10-10 07:11:48 CDT LOG: next MultiXactId: 3599; next
MultiXactOffset: 7197
2007-10-10 07:11:48 CDT LOG: database system was not properly shut
down; automatic recovery in progress
2007-10-10 07:11:48 CDT LOG: redo starts at 70/925E7BB8
nsc 2007-10-10 07:11:48 CDT attbase FATAl: the database system is
starting up
nsc 2007-10-10 07:11:48 CDT attbase FATAl: the database system is
starting up
nsc 2007-10-10 07:11:48 CDT attbase FATAl: the database system is
starting up
nsc 2007-10-10 07:11:48 CDT attbase FATAl: the database system is
starting up
nsc 2007-10-10 07:11:48 CDT attbase FATAl: the database system is
starting up
:
:
:
nsc 2007-10-10 07:11:53 CDT vrzbase FATAl: the database system is
starting up
nsc 2007-10-10 07:11:53 CDT vrzbase FATAl: the database system is
starting up
2007-10-10 07:11:53 CDT LOG: record with zero length at 70/92BBF1C8
2007-10-10 07:11:53 CDT LOG: redo done at 70/92BBF198
nsc 2007-10-10 07:11:53 CDT vrzbase FATAl: the database system is
starting up
nsc 2007-10-10 07:11:53 CDT vrzbase FATAl: the database system is
starting up
nsc 2007-10-10 07:11:53 CDT vrzbase FATAl: the database system is
starting up
2007-10-10 07:11:53 CDT LOG: database system is ready
2007-10-10 07:11:53 CDT LOG: transaction ID wrap limit is 2147484146,
limited by database "postgres"
2007-10-10 07:11:59 CDT LOG: archived transaction log file
"000000010000007000000092"
nsc 2007-10-10 07:19:37 CDT attbase WARNING: there is no transaction in
progress
nsc 2007-10-10 07:20:48 CDT attbase WARNING: there is no transaction in
progress
nsc 2007-10-10 07:23:53 CDT attbase WARNING: there is no transaction in
progress
nsc 2007-10-10 07:24:09 CDT vrzbase LOG: unexpected EOF on client
connection
nsc 2007-10-10 07:24:09 CDT vrzbase LOG: unexpected EOF on client
connection
nsc 2007-10-10 07:24:09 CDT vrzbase LOG: unexpected EOF on client
connection
nsc 2007-10-10 07:24:09 CDT vrzbase LOG: unexpected EOF on client
connection

Thanks
Paramjeet Kaur

-----Original Message-----
From: Bhella Paramjeet-PFCW67
Sent: Wednesday, October 10, 2007 10:43 AM
To: 'pgsql-admin(at)postgresql(dot)org'
Cc: Subbiah Stalin-XCGF84
Subject: Postgres database is not logging any messages after the reboot

Hi

The server that hosts our production postgres databases had to be
rebooted last night as the file systems got remounted as read only.
After the reboot we re-started the databases but now nothing is getting
logged into the pglogs. Can you please assist us in figuring out the
problem and help resolve it. Below listed are the error messages in the
/var/log/messages file due to which we had to reboot the server. This is
a linux x86-64 server. Your prompt response will be highly appreciated.

Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_free_blocks_sb: bit already cleared for b lock 3931168 Oct 10
06:45:02 sandbox kernel: Aborting journal on device sdb1.
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_free_blocks_sb: bit already cleared for b lock 3931169 Oct 10
06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_free_blocks_sb: bit already cleared for b lock 3931170 Oct 10
06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_free_blocks_sb: bit already cleared for b lock 3931171 Oct 10
06:45:02 sandbox kernel: ext3_abort called.
Oct 10 06:45:02 sandbox kernel: EXT3-fs error (device sdb1):
ext3_journal_start_sb: Detected aborted journa l Oct 10 06:45:02 sandbox
kernel: Remounting filesystem read-only Oct 10 06:45:02 sandbox kernel:
EXT3-fs error (device sdb1) in ext3_free_blocks_sb: Journal has aborted
Oct 10 06:45:02 sandbox last message repeated 70 times Oct 10 06:45:02
sandbox kernel: EXT3-fs error (device sdb1) in ext3_reserve_inode_write:
Journal has aborte d Oct 10 06:45:02 sandbox kernel: EXT3-fs error
(device sdb1) in ext3_truncate: Journal has aborted Oct 10 06:45:02
sandbox kernel: EXT3-fs error (device sdb1) in ext3_reserve_inode_write:
Journal has aborte d Oct 10 06:45:02 sandbox kernel: EXT3-fs error
(device sdb1) in ext3_orphan_del: Journal has aborted Oct 10 06:45:02
sandbox kernel: EXT3-fs error (device sdb1) in ext3_reserve_inode_write:
Journal has aborte d Oct 10 06:45:02 sandbox kernel: EXT3-fs error
(device sdb1) in ext3_delete_inode: Journal has aborted Oct 10 06:45:02
sandbox kernel: __journal_remove_journal_head: freeing b_committed_data

Thanks
Paramjeet Bhella