Sudden slow down and spike in system CPU causes max_connections to get exhausted

Hi,

We run postgres 9.1.11, on Centos 6.3, and an ext2 filesystem

Everything will run along okay, and every few hours, for about a couple of minutes, postgres will slow way down. A "select 1" query takes between 10 and 15 seconds to run, and the box in general gets lethargic.

This causes a pile up of connections at the DB, and we run out of max_connections.

This is accompanied with a steep spike in system CPU and load avg. No spike in user CPU or in I/O.

So far:
- We've ruled out check points as a cause.
- We have statement logging turned on and no single statement seems to be causing this. All statements slow down, including "select 1"
- There is no spike in incoming traffic that we can see.

We do typically have a lot of idle connections (1500 connections total, over a 1000 idle at any given time). We're in the midst of installing pgbouncer to try and mitigate the problem, but that still doesn't address the root cause.

Anyone have any tips for why this might be occurring?

Thanks,
Karthik