data on devel code perf dip

I have an example of runs that illustrate a performance
problem that occurred between installing the 7/18 and 8/1
development release codes. I'm running on a PPC64 8-way system,
with 16GB of memory (on a 15GB virtual machine), with a DBT2
workload configured as a 16 warehouse DBT2 with 16 db connections,
no think time. Using Linux 2.6.11/gentoo.

Overview:

Before picture:
run 42 code updated from cvs July 18 12:18pm pacific
metric: 8374.71 NOTPM

After picture (same config params, db size, etc):
run 59 code downloaded from cvs (all of it) Aug1 9:58AM pacific
metric: 4787.61 NOTPM

Oprofile data in 59 (top part copied at the end of this message) but
not in 42 (although the binary is still around and I could in theory
rerun and get it).

I'd like to first get back to the throughput of run 42, and then
remove what's keeping me from getting higher than 45% user with
the original July 18 drop.

************************************************************
Here are more details:

(RUN 42) the previous performance 8374.71 new-order transactions per
minute (NOTPM) and %user around 45% on 8 processors. Even this level of
user utilization is not good, as there is still some bandwidth left in
the log device (not much, but some).

http://developer.osdl.org/maryedie/DBT2_PGSQL/42/

This development code we got on July 18 12:18pm pacific. Tuned it and got
to run 42 on July 26.

(RUN 59) This is an example of a run with code downloaded (whole) on
Aug1 9:58am Pacific. 4787.61 new-order transactions per minute (NOTPM)
and ~22% user over 8 cpus. This has the same config parameters as 42,
with autovacuum off. However, I turned on oprofile for this run. (runs
without oprofile are equivalent).

http://developer.osdl.org/maryedie/DBT2_PGSQL/59/

Top things for postgreSQL in the oprofile are copied below.

CPU: ppc64 POWER5, speed 1656.38 MHz (estimated)
Counted CYCLES events (Processor cycles) with a unit mask of 0x00 (No unit mask) count 100000
samples % app name symbol name
164623113 70.5372 kernel-2.6.11.3 .shared_idle
6641569 2.8458 libc-2.3.4.so (no symbols)
4011874 1.7190 postgres .AllocSetAlloc
2798355 1.1990 postgres .CreateLWLocks
1731687 0.7420 postgres .SearchCatCache
1701936 0.7292 postgres .MemoryContextAllocZeroAligned
1509903 0.6470 postgres .FunctionCall2
1320467 0.5658 postgres .hash_search
1275278 0.5464 oprofiled (no symbols)
1271477 0.5448 postgres .yyparse
1099396 0.4711 postgres .OpernameGetCandidates
794802 0.3406 postgres .expression_tree_walker
731969 0.3136 postgres .MemoryContextAlloc
730266 0.3129 postgres ._bt_compare
709793 0.3041 postgres .DirectFunctionCall1
689405 0.2954 postgres .base_yylex
683776 0.2930 postgres .hash_any
635680 0.2724 postgres .DLMoveToFront
632098 0.2708 kernel-2.6.11.3 .___dst_free
562111 0.2409 kernel-2.6.11.3 .sys_setrlimit
508608 0.2179 postgres .AtEOXact_CatCache
499318 0.2139 postgres .heap_release_fetch
498136 0.2134 kernel-2.6.11.3 .tty_write
494020 0.2117 postgres .XLogInsert

--
Mary Edie Meredith
Initiative Manager
Open Source Development Labs
maryedie(at)hotmail(dot)com
503-906-1942