Don't even bother trying to tune zfs untill after you've tuned postgres,
otherwise your wasting your time.
--
Robert Treat
Build A Brighter LAMP :: Linux Apache {middleware} PostgreSQL
As it turns out, I think the ZFS-on-RAID setup I had is the problem[1]. After running some more I/O tests with "fsync" turned on, I noticed that filesystem "writes" were about 400% slower. Running PostgreSQL without "fsync" made the application run about 1.5 times faster on ZFS versus UFS. So, now it's back to the drawing board to figure out if I can make the synchronous writes faster.
Bob
[1] the suggestions made previously helped a bit, but not nearly enough to overcome the 50% drop originally noted.