backend for database 'A' crashes/is killed -> corrupt index in database 'B'

Please bear with me, as it's taken me a while to be able to reproduce an
issue that is worrisome.

Before I describe the problem, I am using postgresql version 8.4.5 on
CentOS 5.5 (x86_64).

The short version is that if a postgresql backend is killed (by the Linux
OOM handler, or kill -9, etc...) while operations are
taking place in a *different* backend, corruption is introduced in the other
backend. I don't want to say it happens 100% of the time, but it happens
every time I test.

The differences from a stock pg config are primarily to do with
logging and such. The other settings (besides logging
and such) that differ from the stock values include:

wal_sync_method -- I've tried with leaving it as the default, and I've
also tried datasync -- same results.
wal_buffers = 128MB
effective_cache_size = 4GB
work_mem = 1MB
maintenance_work_mem = 1GB

Here is how I am reproducing the problem:

1. Open a psql connection to database A. It may remain idle.
2. Wait for an automated process to connect to database B and start
operations. These operations
3. kill -9 the backend for the psql connection to database A.

Then I observe the backends all shutting down and postgresql entering
recovery mode, which succeeds.
Subsequent operations on other databases appear fine, but not for
database B: An index on one of the tables in database B is corrupted.
It is always the
same index.

2011-03-30 14:51:32 UTC LOG: server process (PID 3871) was terminated by
signal 9: Killed
2011-03-30 14:51:32 UTC LOG: terminating any other active server
processes
2011-03-30 14:51:32 UTC WARNING: terminating connection because of crash
of another server process
2011-03-30 14:51:32 UTC DETAIL: The postmaster has commanded this server
process to roll back the current transaction and exit, because another
server process exited abnormally and possibly corrupted shared memory.
2011-03-30 14:51:32 UTC HINT: In a moment you should be able to reconnect
to the database and repeat your command.
2011-03-30 14:51:32 UTC databaseB databaseB WARNING: terminating connection
because of crash of another server process
2011-03-30 14:51:32 UTC databaseB databaseB DETAIL: The postmaster has
commanded this server process to roll back the current transaction and exit,
because another server process exited abnormally and possibly corrupted
shared memory.
2011-03-30 14:51:32 UTC databaseB databaseB HINT: In a moment you should be
able to reconnect to the database and repeat your command.
2011-03-30 14:51:32 UTC LOG: all server processes terminated;
reinitializing
2011-03-30 14:51:32 UTC LOG: database system was interrupted; last known
up at 2011-03-30 14:46:50 UTC
2011-03-30 14:51:32 UTC databaseB databaseB FATAL: the database system is
in recovery mode
2011-03-30 14:51:32 UTC LOG: database system was not properly shut down;
automatic recovery in progress
2011-03-30 14:51:32 UTC LOG: redo starts at 301/1D328E40
2011-03-30 14:51:33 UTC databaseB databaseB FATAL: the database system is
in recovery mode
2011-03-30 14:51:34 UTC LOG: record with zero length at 301/1EA08608
2011-03-30 14:51:34 UTC LOG: redo done at 301/1EA08558
2011-03-30 14:51:34 UTC LOG: last completed transaction was at log time
2011-03-30 14:51:31.257997+00
2011-03-30 14:51:37 UTC LOG: autovacuum launcher started
2011-03-30 14:51:37 UTC LOG: database system is ready to accept
connections
2011-03-30 14:52:05 UTC databaseB databaseB ERROR: index "<elided>"
contains unexpected zero page at block 0
2011-03-30 14:52:05 UTC databaseB databaseB HINT: Please REINDEX it.

What's more, I can execute a 'DELETE from tableB' (where tableB is the
table that is the one with the troublesome index) without error, but
when I try to *insert* that is when I get a problem. The index is a
standard btree index. The DELETE statement has no where clause.

Other information: The following variables remain commented out and thus
retain their defaults (confirmed with 'show all'):
fsync is on. synchronous_commit is on. full_page_writes is on.
zero_damaged_pages is *off*.

The underlying storage is a software raid1.

'test_fsync' from postgresql 9.0.1 reports:

Loops = 10000

Simple write:
8k write 157205.515/second

Compare file sync methods using one write:
(open_datasync unavailable)
open_sync 8k write 1357.928/second
8k write, fdatasync 3872.508/second
8k write, fsync 3496.659/second

Compare file sync methods using two writes:
(open_datasync unavailable)
2 open_sync 8k writes 1928.987/second
8k write, 8k write, fdatasync 2020.884/second
8k write, 8k write, fsync 2411.653/second

Compare open_sync with different sizes:
open_sync 16k write 3298.716/second
2 open_sync 8k writes 1228.599/second

Test if fsync on non-write file descriptor is honored:
(If the times are similar, fsync() can sync data written
on a different descriptor.)
8k write, fsync, close 2535.765/second
8k write, close, fsync 2162.060/second

My interpretation of these values is that the drives themselves have
their write caches disabled.

Lastly, the problem has been reproduced on at least three totally
different machines.

--
Jon