System overload / context switching / oom, 8.3

pg 8.3.9, Debian Etch, 8gb ram, quadcore xeon, megaraid (more details at end)
~240 active databases, 800+ db connections via tcp.

Everything goes along fairly well, load average from 0.5 to 4.0. Disk
IO is writing about 12-20 MB every 4 or 5 seconds. Cache memory about
4gb. Then under load, we see swapping and then context switch storm and
then oom-killer.

I'm hoping to find some ideas for spreading out the load of bgwriter
and/or autovacuum somehow or possibly reconfiguring memory to help
alleviate the problem, or at least to avoid crashing.

(Hardware/software/configuration specs are below the following dstat output).

I've been able to recreate the context switch storm (without the crash)
by running 4 simultaneous 'vacuum analyze' tasks during a pg_dump.
During these times, htop shows all 8 cpu going red bar 100% for a second
or two or three, and this is when I see the context switch storm.
The following stat data however is from a production workload crash.

During the dstat output below, postgresql was protected by oom_adj -17.
vm_overcommit_memory set to 2, but at this time vm_overcommit_ratio was
still at 50 (has since been changed to 90, should this be 100?). The
memory usage was fairly constant 4056M 91M 3906M, until the end and after
heavier swapping it went to 4681M 984k 3305M (used/buf/cache).

dstat output under light to normal load:
---procs--- ---paging-- -dsk/total- ---system-- ----total-cpu-usage----
run blk new|__in_ _out_|_read _writ|_int_ _csw_|usr sys idl wai hiq siq
0 2 5| 0 0 | 608k 884k| 756 801 | 11 2 83 4 0 0
1 0 4| 0 0 | 360k 1636k|1062 1147 | 13 1 83 2 0 0
2 2 5| 0 0 | 664k 1404k| 880 998 | 13 2 82 4 0 0
0 4 4| 0 0 |2700k 6724k|1004 909 | 10 1 72 16 0 0
0 2 4| 0 0 | 13M 14M|1490 1496 | 13 2 72 12 0 0
1 1 4| 0 0 | 21M 1076k|1472 1413 | 12 2 74 11 0 0
0 3 5| 0 0 | 15M 1712k|1211 1192 | 10 1 76 12 0 0
1 0 4| 0 0 |7384k 1124k|1277 1403 | 15 2 75 9 0 0
0 7 4| 0 0 |8864k 9528k|1431 1270 | 11 2 63 24 0 0
1 3 4| 0 0 |2520k 15M|2225 3410 | 13 2 66 19 0 0
2 1 5| 0 0 |4388k 1720k|1823 2246 | 14 2 70 13 0 0
2 0 4| 0 0 |2804k 1276k|1284 1378 | 12 2 80 6 0 0
0 0 4| 0 0 | 224k 884k| 825 900 | 12 2 86 1 0 0

under heavy load, just before crash, swap use has been increasing for
several seconds or minutes:
---procs--- ---paging-- -dsk/total- ---system-- ----total-cpu-usage----
run blk new|__in_ _out_|_read _writ|_int_ _csw_|usr sys idl wai hiq siq
2 22 9| 124k 28k| 12M 1360k|1831 2536 | 7 4 46 44 0 0
4 7 8| 156k 80k| 14M 348k|1742 2625 | 5 3 53 38 0 0
1 14 7| 60k 232k|9028k 24M|1278 1642 | 4 3 50 42 0 0
0 24 7| 564k 0 | 15M 5832k|1640 2199 | 7 2 41 50 0 0
1 26 7| 172k 0 | 13M 1052k|1433 2121 | 5 3 54 37 0 0
0 15 6| 36k 0 |6912k 35M|1295 3486 | 2 3 58 37 0 0
3 30 2| 0 0 |9724k 13M|1373 2378 | 4 3 48 45 0 0
5 20 4|4096B 0 | 10M 26M|2945 87k | 0 1 44 55 0 0
1 29 8| 0 0 | 19M 8192B| 840 19k | 0 0 12 87 0 0
4 33 3| 0 0 |4096B 0 | 14 39 | 17 17 0 67 0 0
3 31 0| 64k 0 | 116k 0 | 580 8418 | 0 0 0 100 0 0
0 36 0| 0 0 |8192B 0 | 533 12k | 0 0 9 91 0 0
2 32 1| 0 0 | 0 0 | 519 12k | 0 0 11 89 0 0
2 34 1| 0 0 | 16k 0 | 28 94 | 9 0 0 91 0 0
1 32 0| 0 0 | 20k 0 | 467 2295 | 1 0 13 87 0 0
2 32 0| 0 0 | 0 0 | 811 21k | 0 0 12 87 0 0
4 35 3| 0 0 | 44k 0 | 582 11k | 0 0 0 100 0 0
3 37 0| 0 0 | 0 0 | 16 67 | 0 9 0 91 0 0
2 35 0| 0 0 | 0 0 | 519 8205 | 0 2 21 77 0 0
0 37 0| 0 0 | 0 0 | 11 60 | 0 4 12 85 0 0
1 35 1| 0 0 | 20k 0 | 334 2499 | 0 0 23 77 0 0
0 36 1| 0 0 | 80k 0 | 305 8144 | 0 1 23 76 0 0
0 35 3| 0 0 | 952k 0 | 541 2537 | 0 0 16 84 0 0
2 35 2| 0 0 | 40k 0 | 285 8162 | 0 0 24 75 0 0
2 35 0| 100k 0 | 108k 0 | 550 9595 | 0 0 37 63 0 0
0 40 3| 0 0 | 16k 0 |1092 26k | 0 0 26 74 0 0
4 37 3| 0 0 | 96k 0 | 790 12k | 0 0 34 66 0 0
2 39 2| 0 0 | 24k 0 | 77 116 | 8 8 0 83 0 0
2 37 1| 0 0 | 0 0 | 354 2457 | 0 0 29 71 0 0
2 37 0|4096B 0 | 28k 0 |1909 57k | 0 0 27 73 0 0
0 39 1| 0 0 | 32k 0 |1060 25k | 0 0 12 88 0 0
---procs--- ---paging-- -dsk/total- ---system-- ----total-cpu-usage----
run blk new|__in_ _out_|_read _writ|_int_ _csw_|usr sys idl wai hiq siq

SPECS:

PostgreSQL 8.3.9 on i486-pc-linux-gnu, compiled by GCC cc (GCC) 4.1.2
20061115 (prerelease) (Debian 4.1.1-21)
Installed from the debian etch-backports package.

Linux 2.6.18-6-686-bigmem #1 SMP Thu Nov 5 17:30:05 UTC 2009 i686
GNU/Linux (Debian Etch)

8 MB RAM
4 Quad Core Intel(R) Xeon(R) CPU E5440 @ 2.83GHz stepping 06
L1 I cache: 32K, L1 D cache: 32K, L2 cache: 6144K

LSI Logic SAS based MegaRAID driver (batter backed/write cache enabled)
Dell PERC 6/i
# 8 SEAGATE Model: ST973451SS Rev: SM04 (72 GB) ANSI SCSI revision: 05

RAID Configuration:
sda RAID1 2 disks (with pg_xlog wal files on it's own partition)
sdb RAID10 6 disks (pg base dir only)

POSTGRES:

261 databases
238 active databases (w/connection processes)
863 connections to those 238 databases

postgresql.conf:
max_connections = 1100
shared_buffers = 800MB
max_prepared_transactions = 0
work_mem = 32MB
maintenance_work_mem = 64MB
max_fsm_pages = 3300000
max_fsm_relations = 10000
vacuum_cost_delay = 50ms
bgwriter_delay = 150ms
bgwriter_lru_maxpages = 250
bgwriter_lru_multiplier = 2.5
wal_buffers = 8MB
checkpoint_segments = 32
checkpoint_timeout = 5min
checkpoint_completion_target = 0.9
effective_cache_size = 5000MB
default_statistics_target = 100
log_min_duration_statement = 1000
log_checkpoints = on
log_connections = on
log_disconnections = on
log_temp_files = 0
track_counts = on
autovacuum = on
log_autovacuum_min_duration = 0

Thanks for any ideas!
Rob