Re: Re: pg_stat_statements normalisation without invasive changes to the parser (was: Next steps on pg_stat_statements normalisation)

On 03/18/2012 06:12 PM, Peter Geoghegan wrote:
> On 18 March 2012 16:13, Tom Lane<tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> Is there a really strong reason why adequate regression testing isn't
>> possible in a plain-vanilla pg_regress script? A quick look at the
>> script says that it's just doing some SQL commands and then checking the
>> results of queries on the pg_stat_statements views. Admittedly the
>> output would be bulkier in pg_regress, which would mean that we'd not
>> likely want several hundred test cases. But IMO the objective of a
>> regression test is not to memorialize every single case the code author
>> thought about during development. ISTM it would not take very many
>> cases to have reasonable code coverage.
> Hmm. It's difficult to have much confidence that a greatly reduced
> number of test cases ought to provide sufficient coverage. I don't
> disagree with your contention, I just don't know how to judge this
> matter. Given that there isn't really a maintenance burden with
> regression tests, I imagine that that makes it compelling to be much
> more inclusive.
>
> The fact that we rely on there being no concurrent queries might have
> to be worked around for parallel scheduled regression tests, such as
> by doing everything using a separate database, with that database oid
> always in the predicate of the query checking the pg_stat_statements
> view.
>
> I probably would have written the tests in Perl in the first place,
> but I don't know Perl. These tests existed in some form from day 1, as
> I followed a test-driven development methodology, and needed to use a
> language that I could be productive in immediately. There is probably
> no reason why they cannot be re-written in Perl, but spit out
> pg_regress tests, compacting the otherwise-verbose pg_regress input.
> Should I cut my teeth on Perl by writing the tests to do so? How might
> this be integrated with the standard regression tests, if that's
> something that is important?

A pg_regress script doesn't require any perl. It's pure SQL.

It is perfectly possible to make a single test its own group in a
parallel schedule, and this is done now for a number of cases. See
src/test/regress/parallel_schedule. Regression tests run in their own
database set up for the purpose. You should be able to restrict the
regression queries to only the current database.

If you want to generate the tests using some tool, then use whatever
works for you, be it Python or Perl or Valgol, but ideally what is
committed (and this what should be in your patch) will be the SQL output
of that, not the generator plus input. Tests built that way get
automatically run by the buildfarm. Tests that don't use the standard
testing framework don't. You need a *really* good reason, therefore, not
to do it that way.

cheers

andrew