Re: Test code is worth the space

From: Simon Riggs <simon(at)2ndQuadrant(dot)com>
To: Robert Haas <robertmhaas(at)gmail(dot)com>
Cc: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, Peter Geoghegan <pg(at)heroku(dot)com>, Greg Stark <stark(at)mit(dot)edu>, Noah Misch <noah(at)leadboat(dot)com>, Simon Riggs <simon(at)2ndquadrant(dot)com>, PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Test code is worth the space
Date: 2015-08-14 11:47:49
Message-ID: CANP8+j+bMMb457qKiNzvKGXxue-7g8j=f6cOFp_+yMRx3xio3A@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On 13 August 2015 at 00:31, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:

> On Wed, Aug 12, 2015 at 7:20 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> > FWIW, I've objected in the past to tests that would significantly
> > increase the runtime of "make check", unless I thought they were
> > especially valuable (which enumerating every minor behavior of a
> > feature patch generally isn't IMO). I still think that that's an
> > important consideration: every second you add to "make check" is
> > multiplied many times over when you consider how many developers
> > run that how many times a day.
> >
> > We've talked about having some sort of second rank of tests that
> > people wouldn't necessarily run before committing, and that would
> > be allowed to eat more time than the core regression tests would.
> > I think that might be a valuable direction to pursue if people start
> > submitting very bulky tests.
>
> Maybe. Adding a whole new test suite is significantly more
> administratively complex, because the BF client has to get updated to
> run it. And if expected outputs in that test suite change very often
> at all, then committers will have to run it before committing anyway.
>
> The value of a core regression suite that takes less time to run has
> to be weighed against the possibility that a better core regression
> suite might cause us to find more bugs before committing. That could
> easily be worth the price in runtime.

Seems like a simple fix. We maintain all regression tests in full, but keep
slow tests in separate files accessed only by a different schedule.

make check == fast-parallel_schedule

make check-full == parallel_schedule

Probably easier to make one schedule call the other, so we don't duplicate
anything.

Tom gets his fast schedule, others get their full schedule.

--
Simon Riggs http://www.2ndQuadrant.com/
<http://www.2ndquadrant.com/>
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Heikki Linnakangas 2015-08-14 12:35:45 Re: Configure checks and optimizations/crc32 tests
Previous Message Magnus Hagander 2015-08-14 10:37:10 Re: commitfest app bug/feature