Seeing context switch storm with 10/13 snapshot of 8.1beta3

I've been having this problem since trying to upgrade from 7.4.1 to 8.03, and
now 8.1.

It's a dual Xenon machine:
Linux annette.stortek.com 2.4.22-26mdkenterprise #1 SMP Wed Jan 7 07:10:39 MST
2004 i686 unknown unknown GNU/Linux

PostgreSQL version is:
-------------------------------------------------------------------------------
----------------------------
PostgreSQL 8.1beta3 on i686-pc-linux-gnu, compiled by GCC gcc (GCC) 3.3.1
(Mandrake Linux 9.2 3.3.1-2mdk)

[creagrs(at)annette creagrs]$ /usr/local/pgsql810/bin/pg_config
BINDIR = /usr/local/pgsql810/bin
DOCDIR = /usr/local/pgsql810/doc
INCLUDEDIR = /usr/local/pgsql810/include
PKGINCLUDEDIR = /usr/local/pgsql810/include
INCLUDEDIR-SERVER = /usr/local/pgsql810/include/server
LIBDIR = /usr/local/pgsql810/lib
PKGLIBDIR = /usr/local/pgsql810/lib
LOCALEDIR =
MANDIR = /usr/local/pgsql810/man
SHAREDIR = /usr/local/pgsql810/share
SYSCONFDIR = /usr/local/pgsql810/etc
PGXS = /usr/local/pgsql810/lib/pgxs/src/makefiles/pgxs.mk
CONFIGURE = '--enable-syslog' '--prefix=/usr/local/pgsql810' '--enable-debug'
CC = gcc
CPPFLAGS = -D_GNU_SOURCE
CFLAGS = -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Winline -Wendif-labels
-fno-strict-aliasing -g
CFLAGS_SL = -fpic
LDFLAGS = -Wl,-rpath,/usr/local/pgsql810/lib
LDFLAGS_SL =
LIBS = -lpgport -lz -lreadline -lncurses -lcrypt -lresolv -lnsl -ldl -lm -lbsd
VERSION = PostgreSQL 8.1beta3

I can reproduce this problem quite quickly with 20 concurrent queries (< 20
minutes with 5 sets of my app at 4 queries a piece, probably quicker). The
queries nominally take < .1 second, but when all 5 sets run, the queries are up
around 5-6 seconds. CS hit over 100k. Below is a vmstat over a few seconds
which show the problem, then it subsides, then it hits again. Note that when my
application gets up to 5 sets of 8, it never comes down again, and the queries
are over 40 seconds to complete.

When run against 7.4.1, the CS problem does not occur (stays < 10k).

[creagrs(at)annette creagrs]$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
10 0 1032 63120 48 1608472 0 0 5 18 3 6 92 8 0 0
8 0 1032 62584 48 1608480 0 0 0 24 173 97793 39 61 0 0
9 0 1032 62332 48 1608500 0 0 0 32 197 83089 43 57 0 0
4 0 1032 61216 48 1608508 0 0 0 16 158 79967 43 57 0 0
6 0 1032 65208 48 1608556 0 0 0 88 262 91266 52 48 0 0
5 0 1032 67368 48 1608556 0 0 0 292 235 14234 87 13 0 0
4 0 1032 65780 48 1608580 0 0 0 40 173 325 95 5 0 0
4 0 1032 64764 48 1608580 0 0 0 0 144 86 98 2 0 0
6 0 1032 65800 48 1608580 0 0 0 24 197 397 96 4 0 0
6 0 1032 66284 48 1608604 0 0 0 64 234 9284 83 17 0 0
6 0 1032 66104 48 1608604 0 0 0 116 253 87658 56 44 0 0
5 0 1032 66276 48 1608612 0 0 0 56 245 20648 81 19 0 0
6 0 1032 65736 48 1608644 0 0 0 88 250 19699 82 18 0 0
5 0 1032 65316 48 1608652 0 0 0 40 215 73709 70 30 0 0
5 0 1032 67664 48 1608700 0 0 0 96 272 22222 81 19 0 0
4 0 1032 66700 48 1608700 0 0 0 84 188 351 93 7 0 0
4 0 1032 67696 48 1608732 0 0 0 64 210 505 91 9 0 0
4 0 1032 67696 48 1608732 0 0 0 0 139 86 99 1 0 0
4 0 1032 64612 48 1608732 0 0 0 0 134 74 99 1 0 0
5 0 1032 65344 48 1608756 0 0 0 48 206 421 94 6 0 0
4 0 1032 66636 48 1608764 0 0 0 16 171 232 96 4 0 0
4 0 1032 67404 48 1608764 0 0 0 0 136 71 100 0 0 0
7 0 1032 63544 48 1608780 0 0 0 56 239 67250 65 35 0 0
6 0 1032 65168 48 1608796 0 0 0 56 230 87782 59 41 0 0
7 0 1032 66588 48 1608820 0 0 0 72 236 106591 42 58 0 0
8 0 1032 62544 48 1608836 0 0 0 184 244 129696 48 52 0 0
4 0 1032 63504 48 1608860 0 0 0 72 245 110887 48 52 0 0
8 0 1032 63400 48 1608868 0 0 0 16 156 101638 35 65 0 0
8 0 1032 62744 48 1608868 0 0 0 24 193 91159 40 60 0 0
8 0 1032 63700 48 1608868 0 0 0 16 172 85165 42 58 0 0
7 0 1032 62888 48 1608876 0 0 0 48 228 94902 42 58 0 0
4 0 1032 64184 48 1608908 0 0 0 72 234 117585 46 54 0 0
7 0 1032 61832 48 1608948 0 0 0 72 209 104287 46 54 0 0
8 0 1032 64468 48 1608956 0 0 0 56 237 112181 44 56 0 0
7 0 1032 63032 48 1608972 0 0 0 56 236 114977 47 53 0 0
8 0 1032 63204 48 1608980 0 0 0 40 205 107433 45 55 0 0
7 0 1032 63832 48 1608996 0 0 0 56 225 113536 40 60 0 0

Help?

Thanks,
Rob

--
Robert Creager
Advisory Software Engineer
Data Management Group
Sun Microsystems
Robert(dot)Creager(at)Sun(dot)com
303.673.2365 Office
888.912.4458 Pager