Data corruption after SAN snapshot

Hi All,

I have a pretty strange issue that I'm looking for ideas on.
I'm using Postgres Plus Advanced Server 9.1, but I believe this problem is
relevant to Postgres Community. It is certainly possible to be a EDB bug
and I am already working with them on this.

We are migrating 1TB+ from Oracle to PPAS. Our new environment consists of
a primary server with two "read-only" clones. We use NetApps SAN storage
and execute a NetApps consistent snapshot on the primary server and then
use flex clones for the read-only servers. The clones power up, do a short
recovery and all should be well. We have been doing this method for two
years except using PPAS 8.4 and physical servers and ext4.
The new environment is RHEL 6.x guests running inside Redhat Virtualization
using XFS and LVM.

The problem is that after the data load, we take a warm snapshot and the
cloned database are coming up corrupt.
A classic example is: ERROR: could not read block 1 in file
"base/18511/13872": read only 0 of 8192 bytes. Looking at the data file, it
is 8k in size, so obviously we are missing block 1 from the file. So far I
identified indexes and sequences as corrupt, but I believe it could be any
object.
Since the snapshot is essentially a crash, this system is not crash
resistant either.

Looking through the timeline of events, it is clear that data exists in RAM
on the primary server, but is not being written out to the SAN for the
snapshot and hence is missing when the clone starts up. My first thought is
that fsync is not working. PPAS has fsync on and is using fdatasync.

I run a rudimentary test using: dd if=/dev/zero of=dd_test2 bs=8k
count=128k conv=fdatasync and crash the server immediate after dd completes.
Everything behaves as expected. with fsync or fdatasync, the file exists
after the crash and reboot. Leaving out the sync results in a missing file
after the crash/reboot, but that is expected. This simple tests shows that
fdatasync is working, but does not prove this under load.

So, at this point, I don't know if the fdatasync is being issued, but not
honored by the OS stack, or if PPAS is even issuing the sync at all.

Anyone have a solid method to test if fdatasync is working correctly or
thoughts on troubleshooting this? It is extremely time consuming to
replicate the problem, but even then the corruption moves around, so it's
hard to know immediately if there is corruption at all. I'm hoping to
utilize a tool set outside of Postgres to positively eliminate the OS stack.

Sorry for the lengthy post, but hopefully it's clear what is going on.

Thanks!
T