Re: ODBC Driver Failure- MS Access- Large record volume

Hi Greg,

Thanks so much for your response. It's nice to get a dialog going. To
answer your questions:

>1. What version of the ODBC driver are you using? (I found my version
>using REGEDIT to key
>HKEY_CURRENT_USER\Software\Microsoft\Installer\Products)

08.00.0102
I had the same problem running a couple of different versions in the 7.0
series.

>2. What version of Access are you using?

MS Access 2003 SP1

>3. What Access library are you using -DAO or ADO?

Usually DAO, but I've had the same problem using ODBCDirect, repeatedly
executing SQL lines (such as inserts) several million times.

>4. Since Access is typically interactive, how are you transacting these
>millions of records, reading a file or is some external device driving Access?

I'm reading text files, parsing the lines, loading variables, and either
processing AddNew's on a linked table, or doing Inserts in an ODBCDirect
workspace

>5. Are you connecting though the Access objects (normal queries and
>tables), or using pass-through?

Both, as above.

>6. What version of Postgres server are you using? Linux or Windows?

8.0.3 under Windows XP, unfortunately. (Our lab is Linux-deficient)

>7. Is logging turned on at the server? (e.g. pg_ctl start -l my_pg.log)

Not yet. I will activate it. Can you give me a hint on what to look for
in the log?

I will experiment with getting a hold of the connection object and closing
and opening it to start a new driver instance. It takes a long time of
stuffing data through the driver to re-create the crash, so it's hard to
know when I've gotten around the bug or not.

Please let me know if my answers invite any new information.

Oh, also, the link from the Postgres web site has changed, and I'm not sure
where to get new versions of the driver. Any help with my confusion?

With thanks,
Robin

At 02:29 PM 9/8/2005, you wrote:
>Hello and welcome Robin,
>I find the key to getting help is giving enough information.
>
>You might answer a few questions.
>1. What version of the ODBC driver are you using? (I found my version
>using REGEDIT to key HKEY_CURRENT_USER\Software\Microsoft\Installer\Products)
>2. What version of Access are you using?
>3. What Access library are you using -DAO or ADO?
>4. Since Access is typically interactive, how are you transacting these
>millions of records, reading a file or is some external device driving Access?
>5. Are you connecting though the Access objects (normal queries and
>tables), or using pass-through?
>6. What version of Postgres server are you using? Linux or Windows?
>7. Is logging turned on at the server? (e.g. pg_ctl start -l my_pg.log)
>
>
>By the way other list memebers, after installation is there a recommended
>way to determine the pgODBC driver version?
>
>Robin,
>Before knowing any of these answers I can say
>the ADO Connection object is your means of "opening and closing" the ODBC
>(or OLE/DB) driver. Open and close often is the rule of thumb for
>interacting with server database. There is typically caching going on for
>recent connections, so the opening a new connection is not the resource
>hog it might seem. If all that is ok, you may indeed have a pgODBC driver
>version with a leak.
>
>And as a complete aside because it has bitten me too many times, set the
>Jet ConnectionTimeout under
>HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Jet\4.0\Engines\Jet 4.0\ to 0 (never
>timeout). A jet connection (linked table) will loose connectivity after a
>default time of 60 minutes of inactivity unless this key is set to some
>other value (longer or shorter) or 0.
>
>Refreshing linked tables is also do-able. You should be able to google it
>or write back and someone will no doubt expound upon it. The Access
>(version) Handbook's from Sybex are excellent references for the mundane
>to the arcane.
>
>
>
>Robin Weber wrote:
>
>>Hello Community,
>>I'm new to this list.
>>I am a user of the Postgres ODBC driver, mainly under MS Access and
>>Matlab. We deal in very large tabular datasets. (My largest table is
>>currently 72 million records, and it will grow several times that size
>>before the year is out.
>>The driver fails after several million record inserts (under access).
>>Once failed (with nothing more helpful tan an "ODBC call failed" message
>>box), it will process no further requests. (I believe it will also fail
>>after receiving that much data, as well a sending.)
>>(I would think of a memory leak of some sort, but in my 27 pro years of
>>programming, I have never dealt with the inner workings of an ODBC
>>driver, and I don't know how to get driver error information. This is
>>also my first open source experience, so please forgive my naivete.)
>>I would like to figure out how to fix this problem. I'm willing to help
>>debug, but I need help getting started. How do I go about trying to
>>understand this problem?
>>One thing that might be helpful: Is there a programmatic way to get
>>Access to close and unload, then reopen the ODBC driver without actually
>>quitting Access? I notice that when I start Access "fresh" I can pump
>>another several million records through before failing. If I can make my
>>programs unload then reload the driver every few million records, it
>>might make a good workaround for my problem. Any clues out there?
>>With thanks,
>>Robin
>>
>>--------------------------------------------------
>>Robin Weber
>>Data Manager and Application Developer for
>>Marine Mammals and Birds
>>Tagging of Pacific Pelagics Project
>>University of California Santa Cruz
>>Long Marine Lab - Center for Ocean Health
>>100 Shaffer Road
>>Santa Cruz, CA 95060-5730
>>(831) 459-2691
>>
>>
>>---------------------------(end of broadcast)---------------------------
>>TIP 9: In versions below 8.0, the planner will ignore your desire to
>> choose an index scan if your joining column's datatypes do not
>> match
>

--------------------------------------------------
Robin Weber
Data Manager and Application Developer for
Marine Mammals and Birds
Tagging of Pacific Pelagics Project
University of California Santa Cruz
Long Marine Lab - Center for Ocean Health
100 Shaffer Road
Santa Cruz, CA 95060-5730
(831) 459-2691