Re: Automatic testing of patches in commit fest

Hi,

On 2017-09-18 14:26:53 +1200, Thomas Munro wrote:
> A couple of new experimental features on commitfest.cputube.org:

Yay.

> 2. It'll now dump a gdb backtrace for any core files found after a
> check-world failure (if you can find your way to the build log...).
> Thanks to Andres for the GDB scripting for this!

Scripting that should not be needed, except that tools are generally
crappy :(

> The code coverage reports at codecov.io are supposed to allow
> comparison between a branch and master so you can see exactly what
> changed in terms of coverage, but I ran into a technical problem and
> ran out of time for now to make that happen. But you can still click
> your way through to the commit and see a coverage report for the
> commit diff. In other words you can see which modified lines are run
> by make check-world, which seems quite useful.

Yes, it's definitely already useful. If you check
https://codecov.io/gh/postgresql-cfbot/postgresql/commits you can see
the code coverage for the various CF entries. Both the absolute coverage
and, more interestingly, the coverage of the changed lines. There's some
noticeable difference in coverage between the various entries...

E.g. very little of the new stuff in https://codecov.io/gh/postgresql-cfbot/postgresql/commit/ceaa3dbece3c9b98abcaa28009320fde45a83f88
is exercised.

> There are plenty more things we could stick into this pipeline, like
> LLVM sanitizer stuff, valgrind, Coverity, more compilers, <your idea
> here>... but I'm not sure what things really make sense. I may be
> placing undue importance on things that I personally screwed up
> recently :-D

A lot of these probably are too slow to be practical... I know it's not
fun, but a good bit of that probably is going to be making the UI of the
overview a bit better. E.g. direct links to the build / tests logs from
travis/codecov/..., allowing to filter by failing to apply / failing
tests / ok, etc...

Some of this would be considerably easier if the project were ok with
having a .travis.yml in our sourcetree.

Regards,

Andres