Re: PSA: we lack TAP test coverage on NetBSD and OpenBSD

From: Fabien COELHO <coelho(at)cri(dot)ensmp(dot)fr>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: PostgreSQL Developers <pgsql-hackers(at)lists(dot)postgresql(dot)org>
Subject: Re: PSA: we lack TAP test coverage on NetBSD and OpenBSD
Date: 2019-01-17 09:46:34
Message-ID: alpine.DEB.2.21.1901170659210.7416@lancre
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers


Hello Tom,

> Although we've got a few NetBSD and OpenBSD buildfarm critters,
> none of them are doing --enable-tap-tests. If they were, we'd
> have noticed the pgbench regression tests falling over:
>
> [...]
>
> I am, TBH, inclined to fix this by removing that test case rather
> than teaching it another spelling to accept. I think it's very
> hard to make the case that tests like this one are anything but
> a waste of developer and buildfarm time. When they are also a
> portability hazard, it's time to cut our losses. (I also note
> that this test has caused us problems before, cf 869aa40a2 and
> 933851033.)

I'd rather keep it by simply adding the "|unknown" alternative. 30 years
of programming have taught me that testing limit & error cases is useful,
although you never know when it will be proven so.

Client application coverage is currently abysmal, especially "psql"
despite the many script used for testing (39% of lines, 42% of
functions!), pgbench is under 90%. Generally we really need more tests,
not less. TAP tests are a good compromise because they are not always
run, and ISTM sometimes (i.e. you asked for it) is enough.

I agree that some tests can be useless, but I do not think that it applies
to this one. This test also checks that under a bad option pgbench stops
with an appropriate 1 exit status. Recently a patch updated the exit
status of pgbench in many cases to distinguish between different kind
errors, thus having non-regression in this area was shown to be a good
idea. Moreover, knowing that the exit status on getopt errors is
consistent across platform has value, and knowing that there is some
variability is not uninteresting.

--
Fabien.

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Chris Travers 2019-01-17 09:50:56 Re: Proposal for Signal Detection Refactoring
Previous Message Kyotaro HORIGUCHI 2019-01-17 09:28:48 Re: add_partial_path() may remove dominated path but still in use