Odd corruption issue reported on dba.stackexchange.com, need advice

From: Craig Ringer <ringerc(at)ringerc(dot)id(dot)au>
To: "pgsql-general(at)postgresql(dot)org" <pgsql-general(at)postgresql(dot)org>
Subject: Odd corruption issue reported on dba.stackexchange.com, need advice
Date: 2012-07-19 03:35:05
Message-ID: 50078069.5050501@ringerc.id.au
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

Hi all

I've been following a strange issue on dba.stackexchange.com and I could
use opinions from those more clueful than I on the log files posted there.

http://dba.stackexchange.com/questions/20959/recover-postgresql-database-from-wal-errors-on-startup/20961#comment34356_20961

The short version is that the person did a bulk-load of some PostGIS
data using the osm2pgsql data-loader tool (
http://wiki.openstreetmap.org/wiki/Osm2pgsql) to populate a Pg 9.1
database. This appeared to succeed, but when the server was restarted it
failed to come up, complaining that "WAL contains references to invalid
pages" and "page 1493172 of relation base/16385/477861 was
uninitialized" (for many different pages).

The logs of the shutdown suggest that a backend probably crashed, but
that shouldn't cause the WAL and heap corruption observed by the OP.

It's interesting to observe the presence of both a fast and an immediate
shutdown request in the log.

A trimmed log follows, original linked to in the article above:

LOG: received fast shutdown request
LOG: aborting any active transactions
LOG: received immediate shutdown request
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back
the current transaction and exit, because another server process exited
abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and
repeat your command.
LOG: could not send data to client: Broken pipe
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back
the current transaction and exit, because another server process exited
abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and
repeat your command.
LOG: could not send data to client: Broken pipe

[Note: The posted log of the shutdown ends here. The poster asserts that
this is the complete log and that the following statement is the next
one in their log. Very strange.]

LOG: database system was interrupted; last known up at 2012-07-13
00:15:20 UTC
LOG: database system was not properly shut down; automatic recovery in
progress
LOG: redo starts at 4D/A25FE548
LOG: unexpected pageaddr 4D/6F5C6000 in log file 77, segment 196,
offset 6053888
LOG: redo done at 4D/C45C5278
WARNING: page 1493172 of relation base/16385/477861 was uninitialized
WARNING: page 2247965 of relation base/16385/477861 was uninitialized
WARNING: page 1493172 of relation base/16385/477861 was uninitialized
PANIC: WAL contains references to invalid pages
LOG: startup process (PID 21574) was terminated by signal 6: Aborted
LOG: aborting startup due to startup process failure

--
Craig Ringer

Responses

Browse pgsql-general by date

  From Date Subject
Next Message Craig Ringer 2012-07-19 03:50:41 Re: Segmentation fault
Previous Message Tom Lane 2012-07-19 02:15:38 Re: problem with dblink and "drop owned by"