Re: Performance testing framework..

On Wed, 7 Oct 2009, Michael Renner wrote:

> I haven't thought about result aggregation & rendering/UI part of the whole
> thing so far, so if anyone has some ideas in that direction they'd be very
> much appreciated when the time has come.

What I did in pgbench-tools (now available at http://git.postgresql.org/ )
is just dump all the individual run results, tagged by the name of test
and similar key information, into a database. That lets people aggregate
however they want using SQL, and cleanly separates out the data from the
UI. My current rendering approach just pulls data from there into
columnar text files I feed into gnuplot, but you could easily render with
pretty much anything else instead just by tweaking how you pull things
from the database.

All of the dbt-X tools (also in that git repo) instead use simple columnar
text files (and their memory resident equivalent) to hold everything,
feeding the data through the appropriate Perl libraries to compute stats
and render things. Since it drives me crazy to manipulate tables of data
that could go into a database in a scripting language, I never bought into
that approach. But it's completely viable too for a standalone tool.

I'm not aware of anyone working on the job management side of the
"performance farm" yet, so I don't think you duplicated anybody else's
work. Mark Wong (who manages the dbt stuff) and I have both been doing
work on the benchmark results collection and organization side of things,
but basically one test at a time. The specific piece I've been working on
lately is spawning off system monitoring daemons to collect information
during the test, I think I'm on my 3rd generation of trying to get a
solution I'm happy with to that problem.

--
* Greg Smith gsmith(at)gregsmith(dot)com http://www.gregsmith.com Baltimore, MD