Re: mb and ecpg regression tests

On Thu, May 17, 2007 at 10:14:45PM +0200, Magnus Hagander wrote:
> IIRC, Joachim had started working on the ecpg part. (Which is one of the
> reasons I haven't looked at it myself yet)

> Joachim, did you ever get anywhere with that?

I have ecpg tests somehow running under msvc but there are still some issues.
Here are the problems that I encounter(ed):

a) ecpg uses a program for each test that runs and generates output. Those
programs need to be compiled: in MSVC we would either compile them on the
fly while testing or generate a new project for every single executable.
Since the second option is overkill, Magnus suggested to try compiling them
on the fly with the command line compiler. This is ugly but seems to work.

b) ecpg tests use several directories. This causes several problems, one of
them being the resultmap format to be changed, it currently uses:

float4/i.86-pc-mingw32=...

we might want to change the / to some other character here.

c) ecpg uses 3 different files per test. stderr, stdout and generated
source. There is not only just one file to diff against an expected file.

My approach was to have separate init and test routines for each "module"
(backend regression tests, ecpg or whatever) and share the rest.
The init routine sets up stuff and the test routine knows about the specific
test logic. It generates lists of files to diff. The diffing and reporting
is kept common of course. A problem here are alternative comparison files.
We have the resultmap entries and we have "test_n.out". This has to be
combined with ecpg having 3 files. We might want to use

test-platform.stdout instead of test.stdout and
test_1.stderr instead of test.stderr

I have not yet sorted out this last item. Proposals welcome. I hope that I
can fix it on the weekend or at least by the end of next week and then send
my version to whoever is interested.

Joachim