Re: 8.5 release timetable, again

On Thu, Aug 27, 2009 at 10:11 AM, Tom Lane<tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>> ... That sounds a lot like the definition of a
>> regression test suite.  Of course, we have that already, but it's
>> nowhere near comprehensive.  Maybe we should be looking at an expanded
>> test suite that runs on a time scale of hours rather than seconds.
>
> mysql's got one of those, and I haven't noticed that it's kept their
> defect rate down any.  Hour-long regression suites are the sort of
> thing developers won't run.

Well, I'll run them. And I bet we could get volunteers to provide
machines to run them every night, too, against CVS HEAD. This has
been discussed before, and I wasn't the one who suggested it. I can't
speak to the value (or lack thereof) of mysql's regression test suite
as I know nothing about it, but even if it is completely worthless
that does not prove that a worthwhile test suite can't be constructed.

> Worse, regression suites are necessarily
> designed to exercise only 100%-reproducible behavior.

That is true, but our current testing methodology (hoping the
beta-testers find the bugs) seems not to be completely satisfactory
either. Which brings us to:

>> I don't think that any test suite is going to eliminate the need for
>> beta-testing.
>
> Precisely...
>
> What I'd like to see is some sort of test mechanism for WAL recovery.
> What I've done sometimes in the past (and recently had to fix the tests
> to re-enable) is to kill -9 a backend immediately after running the
> regression tests, let the system replay the WAL for the tests, and then
> take a pg_dump and compare that to the dump gotten after a conventional
> run.  However this is quite haphazard since (a) the regression tests
> aren't especially designed to exercise all of the WAL logic, and (b)
> pg_dump might not show the effects of some problems, particularly not
> corruption in non-system indexes.  It would be worth the trouble to
> create a more specific test methodology.

Yep. I was thinking about this as an area for possible improvement.
I don't immediately have a brilliant idea how to do it. I did have
the idea of creating a loadable C function which would panic the
database. This could be used to crash the database at a particular
point (even in mid-query, with sufficient creativity).

I think we would certainly need some more powerful way of checking
pass/failure than exact-text comparisons on SQL query results. Being
a perl guy, the first thing that occurs to me is to write some kind of
test harness in perl that can issue SQL queries as well as do other
things, but I don't have an exact design mapped out in my head, and
I'm sure there are other viable approaches.

> In short: merely making the tests bigger doesn't impress me in the
> least.  Focused testing on areas we aren't covering at all could be
> worth the trouble.

Well, I wasn't suggesting adding a lot more testing of things that
we're already testing. I was assuming that we would craft the
additional tests to hit areas that we are not now covering well. My
point here is only to what Peter said upthread: we want to be able to
get positive results rather than waiting for "enough" negative results
(whatever that means). To get positive results, you must have a test
suite. While letting beta testers test whatever they want has some
value, testing things we think might be likely hiding places for bugs
(such as WAL recovery) has merit, too. Making those tests
well-organized and easily repeatable is, IMHO, a Good Thing.

...Robert