Re: call for creating test suite

"lothar(dot)behrens(at)lollisoft(dot)de" <lothar(dot)behrens(at)lollisoft(dot)de> writes:

> I think, reverse engeneering the mylog to a description or script in a
> language is very
> difficult to handle and maintain.
>
> I don't know the format of mylog, but I know how difficult it is to
> scan log files - even only
> to pick out some aspects only.

I don't think either it will be easy to base and manage a test suite
on log outputs.

I agree with Lothar on basically everything else he said in the
previous message (except I would replace "peace" with "piece" for
better clarity :-)

IMHO, a test suite should be made of snippets of real code, and be as
close as possible to real ODBC use, for two main reasons:

- introducing distance between the tests and the real applications may
introduce bugs or slightly different behaviours, and you end up
debugging the tests instead of the actual code.

- The more the extra dependencies besides ODBC (on external parsers,
languages, tools), the more the effort to run the suite, the less
people running it/using it/contributing to it. For instance, as Lothar
put it, by parsing postgreSQL files you prevent, say mysql (and
continuent FWIW) ODBC developers to contribute. And you create a
dependency on the format of logs, which are meant to be human-readable
and not stable.

That does imply that an ODBC test suite should be mostly written in
C/C++. At worst some Perl or alike could be used to parse & format the
results of the runs but ideally some C/C++ testing framework would
provide that. We have used the CppUnit framework to do that in Carob
and are mostly satisfied about it. Main drawbacks are the limited
documentation (compared to the numerous possibilities and flexibility)
and the verbosity of the code of tests. You can browse carob testing
code here: <https://forge.continuent.org/scm/?group_id=10>, in
directory "carob/test/"

Similar alternatives to CppUnit do exist.

Moreover, I think we definitely want to give a try to the unixodbc
test suite before doing anything. If not to opt for it and contribute
to it, at least to learn from it.

Cheers,

Marc.