Re: Test code is worth the space

From: Noah Misch <noah(at)leadboat(dot)com>
To: Jim Nasby <Jim(dot)Nasby(at)BlueTreble(dot)com>
Cc: Greg Stark <stark(at)mit(dot)edu>, Simon Riggs <simon(at)2ndquadrant(dot)com>, PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Test code is worth the space
Date: 2015-08-18 05:57:43
Message-ID: 20150818055743.GB2129613@tornado.leadboat.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On Mon, Aug 17, 2015 at 02:04:40PM -0500, Jim Nasby wrote:
> On 8/16/15 8:48 AM, Greg Stark wrote:
> >On Sun, Aug 16, 2015 at 7:33 AM, Noah Misch <noah(at)leadboat(dot)com> wrote:
> >>When I've just spent awhile implementing a behavior change, the test diffs are
> >>a comforting sight. They confirm that the test suite exercises the topic I
> >>just changed. Furthermore, most tests today do not qualify under this
> >>stringent metric you suggest. The nature of pg_regress opposes it.
> >
> >It's not a comforting sight for me. It means I need to change the test
> >results and then of course the tests will pass. So they didn't really
> >test anything and I don't really know whether the changes broke
> >anything. Any test I had to adjust for my change was effectively
> >worthless.
> >
> >This is precisely my point about pg_regress and why it's the reason
> >there's pressure not to have extensive tests. It's useful to have some
> >end-to-end black box tests like this but it's self-limiting. You can't
> >test many details without tying the tests to specific behaviour.
> >
> >I have worked with insanely extensive testing suites that didn't have
> >this problem. But they were unit tests for code that was structured
> >around testability. When the API is designed to be testable and you
> >have good infrastructure for mocking up the environment and comparing
> >function results in a much narrower way than just looking at text
> >output you can test the correctness of each function without tying the
> >test to the specific behaviour.
> >
> >*That* gives a real comfort. When you change a function to behave
> >entirely differently and can still run all the tests and see that all
> >the code that depends on it still operate correctly then you know you
> >haven't broken anything. In that case it actually *speeds up*
> >development rather than slowing it down. It lets newbie developers
> >hack on code where they don't necessarily know all the downstream
> >dependent code and not be paralyzed by fear that they'll break
> >something.
>
> As someone who oversaw a database test suite with almost 500 test files
> (each testing multiple cases), I completely agree. Our early framework was
> actually similar to pg_regress and that worked "OK" up to about 200 test
> files, but it got very unwieldy very quickly. We switched to pgTap and it
> was far easier to work with.

My own position is based on having maintained a pg_regress suite an order of
magnitude larger than that. I don't know why that outcome was so different.

> I suspect any effort to significantly improve Postgres test coverage is
> doomed until there's an alternative to pg_regress.

There is the src/test/perl/TestLib.pm harness.

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Noah Misch 2015-08-18 05:59:30 Re: Raising our compiler requirements for 9.6
Previous Message Pavel Stehule 2015-08-18 05:32:42 Re: jsonb array-style subscripting