Re: call for creating test suite

Thu, Feb 02, 2006 at 04:46:42AM -0800, lothar(dot)behrens(at)lollisoft(dot)de wrote:
> If you talk about a description in a text file, how do you interpret
> that description in an application to do that test ?
>
> In my understanding, each description has a fixed format - eg. a
> language.

What application do you mean? I hope you mean testsuite.

There is ODBC API specification. So we know all SQLXXX call and
it's parameters (input, output, sepcific stmt, maybe some more?).
mylog output isn't ready right now. But it may contains something
like:
SQLXXX(param1,param2,...)
It isn't problematic to parse it with e.g. perl and it is more
readable than ini (it has less limitations).

> You may be able to generate these description data by scanning mylog
> data. But in my mind, the result is like a language a specific parser
> could understand.

Ok.

> Regardles, how you would create the input files - detecting a login
> scenario in mylog, you would create a login test file reflecting that
> scenario.
>
> When did you know, a new scenario has begun in mylog ?

It isn't problematic. BTW there is no problem to replay whole run
with mylog. And the scenario begin at the first line of mylog ;-)
We need to create minimalistic reproduceable application now when
we want hunt the bug. It is the same way with every testsuites.

> I think, reverse engeneering the mylog to a description or script in a
> language is very difficult to handle and maintain.

Why? I see no problem. It gives me more power. I can creata test
from whatever programming language I want. It just easy enable mylog
output and create test text file from it (with some free available
script). Do I miss something important?

I think it's more problematic to rewrite the problem into DLL or
another prog. language with somem restrictions.

> I don't know the format of mylog, but I know how difficult it is to
> scan log files - even only
> to pick out some aspects only.

There is no problem. Unix tools like grep, or using perl is very very
simple. When I know what I need to find.

> Who understands these mylog files ?

I hope I understand these files :-) But I want change the mylog
output to generate better logging when it goes into SQLXXX and
when it returns something from SQLXXX.
BTW mylog output isn't so unreadable ;-)

> If I think about a plugin, I think about extensibility of the test
> suite - regardless of the language of implementation.

I think ODBC specification is fixed. I see no way to extend it.

> And more, if you want to compare a test between different databases,
> who maintains the generation of new tests from foreign logfile formats ?

Good point. I'm unable to answer this. But I see one big problem
with this. I see noone who do this for us :-(
BTW there is maybe some chance to get it with unixODBC or iODBC or
from MS ODBC manager tracing?

Still it is easier for normal user to generate mylog output and
send it to us then writing special application to detect the difference.
The test suite with text files from mylog output could be very
good for replying bugs without having reproduceable application.

> What I mean, is, a programmer who uses ODBC API, may use typical
> scenarious.

"May" - this is the real problem. Each implementation use different
access to ODBC driver. I see differences between VB, C++, BC++, Delphi,
...

> If you use these peaces to test pgSQL against, you can find bugs in the
> implementation of new stuff or unknown bugs that appear while you
> fixing other bugs.

I see no difference here againist text files. I can record all
peaces into text files ;-)

> New features in the ODBC driver should end up in a test for that
> feature, thus extention.

Yes I agree. No problem with text files. No difference.

> I hope to be constructive :-)

I hope so

Luf