Re: PostgreSQL unit tests

On Wed, 22 Feb 2006, Alvaro Herrera wrote:

> Robert Treat wrote:
>
> > You could check into what spikesource has been doing. I believe they mostly
> > just piggyback off of our regression tests for postgresql core, but there
> > might still be something that could be built upon. If you look at this url
> > http://developer.spikesource.com/spikewatch/index.jsp?show=component-results&comp-id=22074
> > the actual success information isnt terribly exciting but the "code coverage"
> > url shows something of more interest. There is more stuff if you dig around a
> > bit.
>
> This can't be right. The report for function coverage shows 100% for
> all utf8_and_*.c files, at the end of the listing. Notice how "C/D
> coverage" (I don't know what it means but I assume it's somehow computed
> per lines of code or something) is 0, which is probably the correct
> result, because our regression tests do not test charset conversions at
> all.
>
> I think the bug may be that they use function names to see what is
> actually tested ...
>
> IIRC Gavin Sherry gave a URL to a test coverage result some centuries
> ago. The only thing that I remember about the result was that it was
> surprinsingly low (IMHO at least).

Yes. Coverage was about 50% from memory. This was coverage resulting from
regression tests.

I previously proposed integrating a unit test framework into PostgreSQL.
Getting started wasn't much fun and I gave up. This is because unit
testing is really suited to a functional programming model, IMHO. Testing
the most complex parts of the postgres backend requires a *lot* of state
to be initialised and we'd really have to stomp all over the backend. I do
think that unit testing of areas such as data types would be useful,
particularly the date/time code and arrays as I consider that area of the
code quite fragile. I wouldn't expect the unit tests to find any bugs.
Rather, it would make it easier, I think, for people (particularly new
comers) to hack on that part of the code with more confidence.

The areas of the backend which do not suit unit testing are usually
associated with lots of state or lots of concurrency - like WAL, buffer
manager and so on. The approaches we have at the moment -- regression
tests, user test, load generators (benchmarks et al) -- do an okay job but
they are a brute force approach. They test the common code path, not the
uncommon one. Approaches used by other projects include sophisticated
static analysis (both of language semantics and appliation semantics, such
as 'function bar should never be call unless function foo has been called
first'); model checking (difficult to implement with limited results) and
once of of patches that introduce disasterous conditions in the code
(like low level functions randomly failing, to check that the callers of
such functions deal with such conditions correctly).

Some basic static analysis has been applied to the backend (Stanford
Checker and coverity, etc), no model checking (as far as I am aware) and
Tom had some good results with the last approach easlier in the year, but
I cannot remember what area of the code he was focusing on.

The problem is, these approaches are either of limited use or time
consuming and niche enough that they cannot be done often or across the
whole tree.

Any other thoughts on this?

Thanks,

Gavin