High context switches occurring

Hi,

One of our PG server is experiencing extreme slowness and there are
hundreds of SELECTS building up. I am not sure if heavy context
switching is the cause of this or something else is causing it.

Is this pretty much the final word on this issue?

http://archives.postgresql.org/pgsql-performance/2004-04/msg00249.php

procs memory swap io system
cpu

r b swpd free buff cache si so bi bo in
cs us sy id wa

2 0 20 2860544 124816 8042544 0 0 0 0 0 0 0
0 0 0

2 0 20 2860376 124816 8042552 0 0 0 24 157 115322 13
10 76 0

3 0 20 2860364 124840 8042540 0 0 0 228 172 120003 12
10 77 0

2 0 20 2860364 124840 8042540 0 0 0 20 158 118816 15
10 75 0

2 0 20 2860080 124840 8042540 0 0 0 10 152 117858 12
11 77 0

1 0 20 2860080 124848 8042572 0 0 0 210 202 114724 14
10 76 0

2 0 20 2860080 124848 8042572 0 0 0 20 169 114843 13
10 77 0

3 0 20 2859908 124860 8042576 0 0 0 188 180 115134 14
11 75 0

3 0 20 2859848 124860 8042576 0 0 0 20 173 113470 13
10 77 0

2 0 20 2859836 124860 8042576 0 0 0 10 157 112839 14
11 75 0

The system seems to be fine on iowait/memory side, except the CPU being
busy with the CS. Here's the top output:

11:54:57 up 59 days, 14:11, 2 users, load average: 1.13, 1.66, 1.52

282 processes: 281 sleeping, 1 running, 0 zombie, 0 stopped

CPU states: cpu user nice system irq softirq iowait idle

total 13.8% 0.0% 9.7% 0.0% 0.0% 0.0% 76.2%

cpu00 12.3% 0.0% 10.5% 0.0% 0.0% 0.1% 76.8%

cpu01 12.1% 0.0% 6.1% 0.0% 0.0% 0.1% 81.5%

cpu02 10.9% 0.0% 9.1% 0.0% 0.0% 0.0% 79.9%

cpu03 19.4% 0.0% 14.9% 0.0% 0.0% 0.0% 65.6%

cpu04 13.9% 0.0% 11.1% 0.0% 0.0% 0.0% 74.9%

cpu05 14.9% 0.0% 9.1% 0.0% 0.0% 0.0% 75.9%

cpu06 12.9% 0.0% 8.9% 0.0% 0.0% 0.0% 78.1%

cpu07 14.3% 0.0% 8.1% 0.0% 0.1% 0.0% 77.3%

Mem: 12081720k av, 9273304k used, 2808416k free, 0k shrd,
126048k buff

4686808k actv, 3211872k in_d, 170240k in_c

Swap: 4096532k av, 20k used, 4096512k free 8044072k
cached

PostgreSQL 7.4.7 on i686-redhat-linux-gnu

Red Hat Enterprise Linux AS release 3 (Taroon Update 5)

Linux vl-pe6650-004 2.4.21-32.0.1.ELsmp

This is a Dell Quad XEON. Hyperthreading is turned on, and I am planning
to turn it off as soon as I get a chance to bring it down.

WAL is on separate drives from the OS and database.

Appreciate any inputs please....

Thanks,
Anjan