C# w/ ODBC, 2.1 million list select gives empty DataSet

Hi everyone,

I've been using the ODBC driver for Windows along with my C# (Framework
2.0) for a few years now and it's been working very nicely.

Until now.

One of my customer's database has grown to considerable size over time,
and in particular, one report he attemps to do in my software gives an
empty report.

Upon investigation, I determined that the SELECT I send to the
postgresql gets processed properly and it probably hands over about 2.1
million rows as the response. After that, I use the "Fill" method as
instructed in the "HOW TO" section in the ODBC documentation to fill the
data into a DataSet object.

After considerable amount of work (almost the same as when I do that
same query in PgAdmin III, about 110 seconds), the DataSet gets
initialized empty, with zero tables in it. No errors, no exceptions.
Everything moves along as if nothing happened.

I just upgraded to the latest ODBC drivers (dated december 2009) and
it's not changing my issue.

Am I hitting a limit of some sort? I played with the "ConnectionTimeout"
values of OdbcCommand and OdbcDataAdapter, didn't seem to change.

Here's a selected sample of the code I use:

OdbcCommand odcQuery = new OdbcCommand(sSQLQuery);

using (OdbcConnection dbcPostGresConnection = new
OdbcConnection(_dpDBSettings.GetConnectionString()))
{
try
{
odcQuery.Connection = dbcPostGresConnection;
dbcPostGresConnection.Open();

switch (emCurrentType)
{
case emSqlQueryType.CREATE:
case emSqlQueryType.DELETE:
case emSqlQueryType.UPDATE:
case emSqlQueryType.INSERT:
case emSqlQueryType.ALTER:
case emSqlQueryType.DROP:
{
_kNumberOfAffectedRecordsFromLastQuery =
odcQuery.ExecuteNonQuery();
break;
}

case emSqlQueryType.SELECT:
{
OdbcDataAdapter oddaDatasetFiller = new
OdbcDataAdapter(odcQuery);
oddaDatasetFiller.Fill(_dsLastResults);

if (_dsLastResults.Tables.Count > 0)
{
_kNumberOfAffectedRecordsFromLastQuery =
_dsLastResults.Tables[0].Rows.Count;
}

break;
}
}

_bLastQuerySucceeded = true;
}
catch... etc all possible catches ahead.

Anyone has a suggestion or ideas? I've searched the web, mailing lists
and documentation as I could, nothing seems to be said about this
particular issue.

One interseting thing I might add, I was previously using a very old
ODBC driver, from 2006. This one caused an exception on the .Fill()
method, which was "ERROR [HY000] Out of memory while reading tuples".
Now I'm assuming that since I upgraded to the new driver, not having
this exception means that I'm not running out of memory anymore, right?

Even more details: with the original 2006 drivers, I could see my
application (the .EXE) climb up to 700MB of private bytes/virtual size
in Task Manager until eventually crashing and then not releasing that
memory. So initially, I was investigating a memory leak issue.

With the new drivers, it still climbs to 700MB or so (makes sense, 2
million rows is quite a bit), and then after the "0 rows" returned,
memory is cleared immediately. Makes sense as well, OBDC doesn't carry
data anymore, dataset is almost null, garbage collector is having a
field day.

Thanks for any suggestions or input from anyone.

Regards,

Laurent Chouinard

SEM Logo <http://www.sem.ca> Laurent Chouinard
laurent(dot)chouinard(at)sem(dot)ca <mailto:laurent(dot)chouinard(at)sem(dot)ca>
SEM inc.
3610 Valiquette, St-Laurent, QC, H4S 1X8
Tel: 514-334-7569 / 1-888-334-7569
Fax: 514-334-5922