Re: Test code is worth the space

On 13 August 2015 at 00:31, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:

> On Wed, Aug 12, 2015 at 7:20 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> > FWIW, I've objected in the past to tests that would significantly
> > increase the runtime of "make check", unless I thought they were
> > especially valuable (which enumerating every minor behavior of a
> > feature patch generally isn't IMO). I still think that that's an
> > important consideration: every second you add to "make check" is
> > multiplied many times over when you consider how many developers
> > run that how many times a day.
> >
> > We've talked about having some sort of second rank of tests that
> > people wouldn't necessarily run before committing, and that would
> > be allowed to eat more time than the core regression tests would.
> > I think that might be a valuable direction to pursue if people start
> > submitting very bulky tests.
>
> Maybe. Adding a whole new test suite is significantly more
> administratively complex, because the BF client has to get updated to
> run it. And if expected outputs in that test suite change very often
> at all, then committers will have to run it before committing anyway.
>
> The value of a core regression suite that takes less time to run has
> to be weighed against the possibility that a better core regression
> suite might cause us to find more bugs before committing. That could
> easily be worth the price in runtime.

Seems like a simple fix. We maintain all regression tests in full, but keep
slow tests in separate files accessed only by a different schedule.

make check == fast-parallel_schedule

make check-full == parallel_schedule

Probably easier to make one schedule call the other, so we don't duplicate
anything.

Tom gets his fast schedule, others get their full schedule.

--
Simon Riggs http://www.2ndQuadrant.com/
<http://www.2ndquadrant.com/>
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services