Re: PSA: we lack TAP test coverage on NetBSD and OpenBSD

From: Fabien COELHO <coelho(at)cri(dot)ensmp(dot)fr>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: PostgreSQL Developers <pgsql-hackers(at)lists(dot)postgresql(dot)org>
Subject: Re: PSA: we lack TAP test coverage on NetBSD and OpenBSD
Date: 2019-01-17 09:46:34
Message-ID: alpine.DEB.2.21.1901170659210.7416@lancre
Views: Raw Message | Whole Thread | Download mbox
Lists: pgsql-hackers

Hello Tom,

> Although we've got a few NetBSD and OpenBSD buildfarm critters,
> none of them are doing --enable-tap-tests. If they were, we'd
> have noticed the pgbench regression tests falling over:
> [...]
> I am, TBH, inclined to fix this by removing that test case rather
> than teaching it another spelling to accept. I think it's very
> hard to make the case that tests like this one are anything but
> a waste of developer and buildfarm time. When they are also a
> portability hazard, it's time to cut our losses. (I also note
> that this test has caused us problems before, cf 869aa40a2 and
> 933851033.)

I'd rather keep it by simply adding the "|unknown" alternative. 30 years
of programming have taught me that testing limit & error cases is useful,
although you never know when it will be proven so.

Client application coverage is currently abysmal, especially "psql"
despite the many script used for testing (39% of lines, 42% of
functions!), pgbench is under 90%. Generally we really need more tests,
not less. TAP tests are a good compromise because they are not always
run, and ISTM sometimes (i.e. you asked for it) is enough.

I agree that some tests can be useless, but I do not think that it applies
to this one. This test also checks that under a bad option pgbench stops
with an appropriate 1 exit status. Recently a patch updated the exit
status of pgbench in many cases to distinguish between different kind
errors, thus having non-regression in this area was shown to be a good
idea. Moreover, knowing that the exit status on getopt errors is
consistent across platform has value, and knowing that there is some
variability is not uninteresting.


In response to


Browse pgsql-hackers by date

  From Date Subject
Next Message Chris Travers 2019-01-17 09:50:56 Re: Proposal for Signal Detection Refactoring
Previous Message Kyotaro HORIGUCHI 2019-01-17 09:28:48 Re: add_partial_path() may remove dominated path but still in use