Lock pileup causes server to stall

All,

pg version: 9.3.5
RHEL 6.5
128GB/32 cores
Configured with shared_buffers=16GB
Java/Tomcat/JDBC application

Server has an issue that whenever we get lock waits (transaction lock
waits, usually on an FK dependancy) lasting over a minute or more than
10 at once, *all* queries on the server slow to a crawl, taking 100X to
400X normal execution times.

Other info:
* This applies even to queries which are against other databases, so
it's not purely a lock blocking issue.
* this database routinely has a LOT of lock conlicts, churning through 1
million multixacts per day
* pgBouncer is also involved in this stack, and may be contributing to
the problem in some way
* at no time is the DB server out of CPU (max usage = 38%), RAM, or
doing major IO (max %util = 22%).
* BIND statements can be slow as well as EXECUTEs.

I don't have full query logs from a stall period yet, so I'll have more
information when I do: for example, is it ALL queries which are slow or
just some of them?

However, I thought this list would have some other ideas where to look.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com