Re: Buildfarm TAP testing is useless as currently implemented

On 07/27/2015 12:15 PM, Andrew Dunstan wrote:
>
> On 07/27/2015 10:06 AM, Tom Lane wrote:
>> I challenge anybody to figure out what happened here:
>> http://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hornet&dt=2015-07-27%2010%3A25%3A17
>>
>> or here:
>> http://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=hamster&dt=2015-07-04%2016%3A00%3A23
>>
>> or here:
>> http://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=crake&dt=2015-07-07%2016%3A35%3A06
>>
>>
>> With no visibility of pg_ctl's output, and no copy of the postmaster
>> log,
>> there is no chance of debugging intermittent failures like this one.
>> This isn't entirely the buildfarm's fault --- AFAICS, prove-based
>> testing
>> has inadequate error reporting by design. If "not ok" isn't enough
>> information for you, tough beans. (It might help if the farm script
>> captured the postmaster log after a failure, but that would do nothing
>> for prove's unwillingness to pass through client-side messages.)
>>
>> I think we should disable TAP testing in the buildfarm until there is
>> some credible form of error reporting for it. I've grown tired of
>> looking into buildfarm failure reports only to meet a dead end.
>> Aside from the wasted investigation time, which admittedly isn't huge,
>> there's an opportunity cost in that subsequent test steps didn't get
>> run.
>
>
>
> Well, it does create a lot of files that we don't pick up. An example
> list is show below, and I am attaching their contents in a single
> gzipped attachment. However, these are in the wrong location. This was
> a vpath build and yet these tmp_check directories are all created in
> the source tree. Let's fix that and then I'll set about having the
> buildfarm collect them. That should get us further down the track.
>

The situation should now be substantially improved. This buildfarm
change
<https://github.com/PGBuildFarm/client-code/commit/e684baacf9cb9f9d821be5088b15b336dc6aae07>
uses today's core changes to pick up log files. See
<http://www.pgbuildfarm.org/cgi-bin/show_stage_log.pl?nm=crake&dt=2015-07-28%2023%3A08%3A54&stg=bin-check>
for an example.

cheers

andrew