Diagnosing Postgres Segfault

Early this morning one of our databases suffered a segfault. The only note
on this in /var/log/messages was:

Nov 29 03:34:07 servername kernel: postmaster[18265]: segfault at
0000000017b82000 rip 0000003c45e7c266 rsp 00007fff2ca1b7d8 error 4

(replaced actual server name with servername)

The postgres log has a bit more information:

2010-11-29 03:34:07 PST LOG: server process (PID 18265) was terminated by
signal 11

2010-11-29 03:34:07 PST LOG: terminating any other active server processes

2010-11-29 03:34:07 PST 10.2.0.18 WARNING: terminating connection because
of crash of another server process

2010-11-29 03:34:07 PST 10.2.0.18 DETAIL: The postmaster has commanded this
server process to roll back the current transaction and exit, because
another server process exited abnormally and possibly corrupted shared
memory.

2010-11-29 03:34:07 PST 10.2.0.18 HINT: In a moment you should be able to
reconnect to the database and repeat your command.

2010-11-29 03:34:07 PST 10.2.0.18 WARNING: terminating connection because
of crash of another server process

2010-11-29 03:34:07 PST 10.2.0.18 DETAIL: The postmaster has commanded this
server process to roll back the current transaction and exit, because
another server process exited abnormally and possibly corrupted shared
memory.

2010-11-29 03:34:07 PST 10.2.0.18 HINT: In a moment you should be able to
reconnect to the database and repeat your command.

2010-11-29 03:34:07 PST 10.2.0.18 WARNING: terminating connection because
of crash of another server process

2010-11-29 03:34:07 PST 10.2.0.18 DETAIL: The postmaster has commanded this
server process to roll back the current transaction and exit, because
another server process exited abnormally and possibly corrupted shared
memory.

2010-11-29 03:34:07 PST 10.2.0.18 HINT: In a moment you should be able to
reconnect to the database and repeat your command.

2010-11-29 03:34:07 PST 10.2.0.18 FATAL: the database system is in recovery
mode

2010-11-29 03:34:07 PST LOG: all server processes terminated;
reinitializing

2010-11-29 03:34:10 PST 10.2.0.17 FATAL: the database system is starting up

2010-11-29 03:34:10 PST 10.2.0.17 FATAL: the database system is starting up

2010-11-29 03:34:10 PST 127.0.0.1 FATAL: the database system is starting up

2010-11-29 03:34:10 PST 10.2.0.17 FATAL: the database system is starting up

2010-11-29 03:34:10 PST 10.2.0.18 FATAL: the database system is starting up

2010-11-29 03:34:10 PST 10.2.0.18 FATAL: the database system is starting up

2010-11-29 03:34:10 PST LOG: database system was interrupted at 2010-11-29
03:30:26 PST

2010-11-29 03:34:10 PST LOG: checkpoint record is at 41F/723A1648

2010-11-29 03:34:10 PST LOG: redo record is at 41F/723A1648; undo record
is at 0/0; shutdown FALSE

2010-11-29 03:34:10 PST LOG: next transaction ID: 0/1144655884; next OID:
630827

2010-11-29 03:34:10 PST LOG: next MultiXactId: 1; next MultiXactOffset: 0

2010-11-29 03:34:10 PST LOG: database system was not properly shut down;
automatic recovery in progress

2010-11-29 03:34:10 PST LOG: record with zero length at 41F/723A1698

2010-11-29 03:34:10 PST LOG: redo is not required

2010-11-29 03:34:10 PST 10.2.0.17 FATAL: the database system is starting up

2010-11-29 03:34:10 PST LOG: database system is ready

I’m trying to figure out what might have caused this, partly because I want
to replicate the segfault to see if it is the cause of another system
randomly freezing on us. Is there any way to figure out exactly what query
being run at the time of the crash was doing? Could the xlog files help me
in anyway?