Re: Re: new tests post-feature freeze (was pgsql: Add TAP tests for pg_dump)

On 5/8/16 11:29 AM, Stephen Frost wrote:
>> My suggestion is that, from this point forward, we add new tests to
>> > 9.6 only if they are closely related to a bug that is getting fixed or
>> > a feature that is new in 9.6. I think that's a reasonable compromise,
>> > but what do others think?
> I'm willing to accept that compromise, but I'm not thrilled with it due
> to what it will mean for the process I'm currently going through. The
> approach I've been using has been to add tests to gain more code
> coverage of the code in pg_dump. That has turned up multiple
> pre-existing bugs in pg_dump but the vast majority of the tests come
> back clean. This compromise would mean that I'd continue to work
> through the code coverage tests, but would have to segregate out and
> commit only those tests which actually reveal bugs, once those bugs have
> been fixed (as to avoid turning the buildfarm red). The rest of the
> tests would still get written, but since they currently don't reveal
> bugs, they would be shelved until development is opened for 9.7.

Having done extensive database unit testing in the past, I've
experienced what Stephen's talking about and agree it's a real pain.
With tap-style tests (or really anything that's not dependent on
something as fragile as diffing output), there's pretty low risk to
adding more tests that are passing.

As for tests distracting people from reviewing patches, robust tests
significantly reduce the need for manual review. I think it's a much
better approach to take a methodical approach of writing tests to help
verify a feature works than just randomly banging on it.
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532) mobile: 512-569-9461