Re: performance-test farm

Tomas Vondra wrote:
> Actually I was not aware of how the buildfarm works, all I
> knew was there's something like that because some of the hackers mention
> a failed build on the mailing list occasionally.
>
> So I guess this is a good opportunity to investigate it a bit ;-)
>
> Anyway I'm not sure this would give us the kind of environment we need
> to do benchmarks ... but it's worth to think of.
>

The idea is that buildfarm systems that are known to have a) reasonable
hardware and b) no other concurrent work going on could also do
performance tests. The main benefit of this approach is it avoids
duplicating all of the system management and source code building work
needed for any sort of thing like this; just leverage the buildfarm
parts when they solve similar enough problems. Someone has actually
done all that already; source code was last sync'd to the build farm
master at the end of March: https://github.com/greg2ndQuadrant/client-code

By far the #1 thing needed to move this forward from where it's stuck at
now is someone willing to dig into the web application side of this.
We're collecting useful data. It needs to now be uploaded to the
server, saved, and then reports of what happened generated. Eventually
graphs of performance results over time will be straighforward to
generate. But the whole idea requires someone else (not Andrew, who has
enough to do) sits down and figures out how to extend the web UI with
these new elements.

> I guess we could run a script that collects all those important
> parameters and then detect changes. Anyway we still need some 'really
> stable' machines that are not changed at all, to get a long-term baseline.
>

I have several such scripts I use, and know where two very serious ones
developed by others are at too. This part is not a problem. If the
changes are big enough to matter, they will show up as a difference on
the many possible "how is the server configured?" reports, we just need
to pick the most reasonable one. It's a small details I'm not worried
about yet.

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us