Re: [COMMITTERS] pgsql: Add TAP tests for pg_dump

On 5/7/16 9:36 AM, Stephen Frost wrote:
> Honestly, over the next couple of months between feature-freeze and
> release, I'd like to add even more tests, and not just to pg_dump but
> also to other commands that don't have very good testing today (psql, in
> particular, but pg_dumpall needs more also, and there's more to do with
> pg_dump too).

If we're going to do that, there will be no stopping it. I also have a
bunch of code in this area lined up, but I was hoping to submit it to
the commit fest process at the right time and order to get feedback and
testing. If we're going to allow new tests being developed right up
until release, I can just stop doing release preparation work right now
and get back to coding and reviewing.

Ultimately, tests are code and should be treated as such.

> When it comes to packaging, if adding tests using the existing test
> infrastructure (the TAP system) causes a problem there, then I think we
> need to address that issue rather than not add new tests. Packagers
> also always have the option to not enable the tap tests, if there really
> is an issue there which can't be addressed.

Well, that's like saying, if the feature I just committed the night
before the release turns out to be problematic, you can just ifdef it
out. I don't think we want that, and I think it simplifies the way the
world works. Because new code interacts with old code, and then there
will be even more new code which interacts with other code, and so it
becomes harder and harder to isolate issues. Yeah, if adding more tests
causes instabilities because of issues not related to the tests
themselves, we should fix that. But that doesn't mean we should
hand-wave past the fact that that potential exists. That's software
development. If everything were perfect, we wouldn't need a beta period
at all.

The configure flag to disable TAP tests isn't there so that we get
license to play with them at random, it's because we wanted to make the
software dependency optional. The psql tests run under pg_regress, so
they can't be made optional anyway.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services