Re: Test code is worth the space

On Fri, Aug 14, 2015 at 12:47:49PM +0100, Simon Riggs wrote:
> On 13 August 2015 at 00:31, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> > On Wed, Aug 12, 2015 at 7:20 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> > > We've talked about having some sort of second rank of tests that
> > > people wouldn't necessarily run before committing, and that would
> > > be allowed to eat more time than the core regression tests would.
> > > I think that might be a valuable direction to pursue if people start
> > > submitting very bulky tests.
> >
> > Maybe. Adding a whole new test suite is significantly more
> > administratively complex, because the BF client has to get updated to
> > run it. And if expected outputs in that test suite change very often
> > at all, then committers will have to run it before committing anyway.
> >
> > The value of a core regression suite that takes less time to run has
> > to be weighed against the possibility that a better core regression
> > suite might cause us to find more bugs before committing. That could
> > easily be worth the price in runtime.
>
> Seems like a simple fix. We maintain all regression tests in full, but keep
> slow tests in separate files accessed only by a different schedule.
>
> make check == fast-parallel_schedule
> make check-full == parallel_schedule

+1 for a split, though I would do "make quickcheck" and "make check". Using
fewer tests should be a conscious decision, and "check" is the widely-known
Makefile target. In particular, non-hackers building production binaries need
the thorough test battery. (As a bonus, the buildfarm wouldn't miss a beat.)