Re: performance-test farm

On Mon, Mar 5, 2012 at 5:20 PM, Tomas Vondra <tv(at)fuzzy(dot)cz> wrote:
>> The idea is that buildfarm systems that are known to have a) reasonable
>> hardware and b) no other concurrent work going on could also do
>> performance tests.  The main benefit of this approach is it avoids
>> duplicating all of the system management and source code building work
>> needed for any sort of thing like this; just leverage the buildfarm
>> parts when they solve similar enough problems.  Someone has actually
>> done all that already; source code was last sync'd to the build farm
>> master at the end of March:  https://github.com/greg2ndQuadrant/client-code
>>
>> By far the #1 thing needed to move this forward from where it's stuck at
>> now is someone willing to dig into the web application side of this.
>> We're collecting useful data.  It needs to now be uploaded to the
>> server, saved, and then reports of what happened generated.  Eventually
>> graphs of performance results over time will be straighforward to
>> generate.  But the whole idea requires someone else (not Andrew, who has
>> enough to do) sits down and figures out how to extend the web UI with
>> these new elements.
>
> Hi,
>
> I'd like to revive this thread. A few days ago we have finally got our
> buildfarm member working (it's called "magpie") - it's spending ~2h a
> day chewing on the buildfarm tasks, so we can use the other 22h to do
> some useful work.
>
> I suppose most of you are busy with 9.2 features, but I'm not so I'd
> like to spend my time on this.
>
> Now that I had to set up the buildfarm member I'm somehow aware of how
> the buildfarm works. I've checked the PGBuildFarm/server-code and
> greg2ndQuadrant/client-code repositories and while I certainly am not a
> perl whiz, I believe I can tweak it to handle the performance-related
> result too.
>
> What is the current state of this effort? Is there someone else working
> on that? If not, I propose this (for starters):
>
>  * add a new page "Performance results" to the menu, with a list of
>    members that uploaded the perfomance-results
>
>  * for each member, there will be a list of tests along with a running
>    average for each test, last test and indicator if it improved, got
>    worse or is the same
>
>  * for each member/test, a history of runs will be displayed, along
>    with a simple graph

I suspect that the second of these is not that useful; presumably
there will be many small, irrelevant variations. The graph is the key
thing.

> I'm not quite sure how to define which members will run the performance
> tests - I see two options:
>
>  * for each member, add a flag "run performance tests" so that we can
>    choose which members are supposed to be safe
>
>  OR
>
>  * run the tests on all members (if enabled in build-farm.conf) and
>    then decide which results are relevant based on data describing the
>    environment (collected when running the tests)

First option seems better to me, but I don't run any buildfarm critters, so...

> I'm also wondering if
>
>  * using the buildfarm infrastructure the right thing to do, if it can
>    provide some 'advanced features' (see below)
>
>  * we should use the current buildfarm members (although maybe not all
>    of them)

The main advantage of using the buildfarm, AFAICS, is that it would
make it less work for people to participate in this new thing.

>  * it can handle one member running the tests with different settings
>    (various shared_buffer/work_mem sizes, num of clients etc.) and
>    various hw configurations (for example magpie contains a regular
>    SATA drive as well as an SSD - would be nice to run two sets of
>    tests, one for the spinner, one for the SSD)

That would be nice.

>  * this can handle 'pushing' a list of commits to test (instead of
>    just testing the HEAD) so that we can ask the members to run the
>    tests for particular commits in the past (I consider this to be
>    very handy feature)

That would be nice, too.

I think it's great that you want to put some energy into this. It's
something we have been talking about for years, but if we're making
any progress at all, it's glacial.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company