Re: Server Down problem

Volker <arendt(at)wiwi(dot)uni-wuppertal(dot)de> writes:
> currently we have a server down problem. The server is a IBM xSeries
> 306, 1GB RAM, HD 36GB RAID1. Operating System ist Debian Sarge 3.1.
> Postgresql version is 7.4.7.
> The reason for this ist not knwon. The postmaster won't come up any
> more. Here is the relevant content of the postgres.log file:

It looks to me like pg_control is quite out of sync with the files in
pg_xlog. pg_control claims to have checkpointed as recently as 16-Aug
but there is no file newer than 7-Aug in pg_xlog. The other thing that
is strange is that the filename numbers in pg_xlog correspond to WAL
locations much higher than what pg_control claims is the end of WAL.

Is $PGDATA kept on a dismountable volume (ie, not the root disk)?
If so it might be a good idea to unmount the volume and look to see
if there's anything under the mount point. I recall having seen
corruption that came from trying to start Postgres before the $PGDATA
disk had been mounted --- the initscript happily initdb'd a new database
under the mount-point directory, and then when the main disk did come up
things were badly hosed because the server was working with a pg_control
in memory that was completely out of sync with everything else. This
looks a bit like that might have happened here.

As far as running pg_resetxlog goes, there is advice about setting the
parameters in recent releases' documentation; try
http://developer.postgresql.org/docs/postgres/app-pgresetxlog.html
for the latest. (Some of the parameters mentioned don't exist in
7.4; just ignore 'em.) I'm afraid though that you may have actual
database corruption. If so pg_resetxlog won't fix it.

regards, tom lane