Re: Recording test runtimes with the buildfarm

On Thu, 11 Jun 2020 at 10:02, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>
> Thomas Munro <thomas(dot)munro(at)gmail(dot)com> writes:
> > I've been doing that in a little database that pulls down the results
> > and analyses them with primitive regexes. First I wanted to know the
> > pass/fail history for each individual regression, isolation and TAP
> > script, then I wanted to build something that could identify tests
> > that are 'flapping', and work out when the started and stopped
> > flapping etc. I soon realised it was all too noisy, but then I
> > figured that I could fix that by detecting crashes. So I classify
> > every top level build farm run as SUCCESS, FAILURE or CRASH. If the
> > top level run was CRASH, than I can disregard the individual per
> > script results, because they're all BS.
>
> If you can pin the crash on a particular test script, it'd be useful
> to track that as a kind of failure. In general, though, both crashes
> and non-crash failures tend to cause collateral damage to later test
> scripts --- if you can't filter that out then the later scripts will
> have high false-positive rates.

I guess the fact that you've both needed to do analysis on individual
tests shows that there might be a call for this beyond just recording
the test's runtime.

If we had a table that stored the individual test details, pass/fail
and just stored the timing information along with that, then, even if
the timing was unstable, it could still be useful for some analysis.
I'd be happy enough even if that was only available as a csv file
download. I imagine the buildfarm does not need to provide us with
any tools for doing analysis on this. Ideally, there would be some
run_id that we could link it back to the test run which would give us
the commit SHA, and the animal that it ran on. Joining to details
about the animal could be useful too, e.g perhaps a certain test
always fails on 32-bit machines.

I suppose that maybe we could modify pg_regress to add a command line
option to have it write out a machine-readable file, e.g:
testname,result,runtime\n, then just have the buildfarm client ship
that off to the buildfarm server to record in the database.

David