Re: Adding a pgbench run to buildfarm

On Mon, 24 Jul 2006, Mark Kirkwood wrote:

> Tom Lane wrote:
> > Mark Kirkwood <markir(at)paradise(dot)net(dot)nz> writes:
> >> Scale factor 10 produces an accounts table of about 130 Mb. Given that
> >> most HW these days has at least 1G of ram, this probably means not much
> >> retrieval IO is tested (only checkpoint and wal fsync). Do we want to
> >> try 100 or even 200? (or recommend scale factor such that size > ram)?
> >
> > That gets into a different set of questions, which is what we want the
> > buildfarm turnaround time to be like. The faster members today produce
> > a result within 10-15 minutes of pulling their CVS snaps, and I'd be
> > seriously unhappy if that changed to an hour or three. Maybe we need to
> > divorce compile/regression tests from performance tests?
> >
> >
>
> Right - this leads to further questions like, what the performance
> testing on the buildfarms is actually for. If it is mainly to catch
> regressions introduced by any new code, then scale factor 10 (i.e
> essentially in memory testing) may in fact be the best way to show this up.

It introduces a problem though. Not all machines stay the same over time.
A machine may by upgraded, a machine may be getting backed up or may in
some other way be utilised during a performance test. This would skew the
stats for that machine. It may confuse people more than help them...

At the very least, the performance figures would need to be accompanied by
details of what other processes were running and what resources they were
chewing during the test.

This is what was nice about the OSDL approach. Each test was preceeded by
an automatic reinstall of the OS and the machines were specifically for
testing. The tester had complete control.

We could perhaps mimic some of that using virtualisation tools which
control access to system resources but it wont work on all platforms. The
problem is that it probably introduces a new variable, in that I'm not
sure that virtualisation software can absolutely limit CPU resources a
particular container has. That is, you might not be able to get
reproducible runs with the same code. :(

Just some thoughts.

Thanks,

Gavin