Skip site navigation (1) Skip section navigation (2)

Re: Synchronisation problems in COPY IN

From: Kim Bisgaard <kim+pg(at)alleroedderne(dot)adsl(dot)dk>
To: pgsql-jdbc(at)postgresql(dot)org
Subject: Re: Synchronisation problems in COPY IN
Date: 2010-09-16 11:31:34
Message-ID: 4C920016.5000206@alleroedderne.adsl.dk (view raw or flat)
Thread:
Lists: pgsql-jdbc
Hi again,

We have corrected to program to wait for the thread to die before 
closing the db-connection - see attached source.

But we still see problems related to overwritten data buffer. See below 
for errors using build 800 or 900 driver - the webpage is a bit 
uncertain ;-)

Any ideas?

Thanks in advance!
Kim

Sep 15, 2010 6:31:40 PM dk.dmi.appl.evejrfeeder.EvejrDB writeIndexedPointData
INFO: Inserting data in temporary_temperature_over_sea_2_m_3 at valid time 2010-09-15 12:00:00 for 38882 gridpoints
Sep 15, 2010 6:31:40 PM dk.dmi.appl.evejrfeeder.EvejrDBCopy run
INFO: ERROR: extra data after last expected column
   Where: COPY temporary_temperature_over_sea_2_m_3, line 186: "60000187,2010-09-15 12:00:00,288.10498046860000746,2010-09-15 12:00:00,287.8037109375"
Sep 15, 2010 6:31:40 PM dk.dmi.appl.evejrfeeder.EvejrDBCopy run
INFO: ERROR: extra data after last expected column
   Where: COPY temporary_temperature_over_sea_2_m_3, line 186: "60000187,2010-09-15 12:00:00,288.10498046860000746,2010-09-15 12:00:00,287.8037109375"
org.postgresql.util.PSQLException: ERROR: extra data after last expected column
   Where: COPY temporary_temperature_over_sea_2_m_3, line 186: "60000187,2010-09-15 12:00:00,288.10498046860000746,2010-09-15 12:00:00,287.8037109375"
	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2062)
	at org.postgresql.core.v3.QueryExecutorImpl.processCopyResults(QueryExecutorImpl.java:929)
	at org.postgresql.core.v3.QueryExecutorImpl.writeToCopy(QueryExecutorImpl.java:838)
	at org.postgresql.core.v3.CopyInImpl.writeToCopy(CopyInImpl.java:53)
	at org.postgresql.copy.CopyManager.copyIn(CopyManager.java:179)
	at org.postgresql.copy.CopyManager.copyIn(CopyManager.java:161)
	at dk.dmi.appl.evejrfeeder.EvejrDBCopy.run(Unknown Source)
	at java.lang.Thread.run(Thread.java:619)
Sep 15, 2010 6:31:43 PM dk.dmi.appl.evejrfeeder.EvejrDB writeIndexedPointData



Sep 15, 2010 6:52:21 PM dk.dmi.appl.evejrfeeder.EvejrDB writeIndexedPointData
INFO: Inserting data in temporary_wind_speed_150_m_3 at valid time 2010-09-15 12:00:00 for 38882 gridpoints
Sep 15, 2010 6:52:21 PM dk.dmi.appl.evejrfeeder.EvejrDBCopy run
INFO: ERROR: missing data for column "wind_speed_150_m"
   Where: COPY temporary_wind_speed_150_m_3, line 173: "60000188,2010-09-15 12:00:00"
Sep 15, 2010 6:52:21 PM dk.dmi.appl.evejrfeeder.EvejrDBCopy run
INFO: ERROR: missing data for column "wind_speed_150_m"
   Where: COPY temporary_wind_speed_150_m_3, line 173: "60000188,2010-09-15 12:00:00"
org.postgresql.util.PSQLException: ERROR: missing data for column "wind_speed_150_m"
   Where: COPY temporary_wind_speed_150_m_3, line 173: "60000188,2010-09-15 12:00:00"
	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2062)
	at org.postgresql.core.v3.QueryExecutorImpl.processCopyResults(QueryExecutorImpl.java:929)
	at org.postgresql.core.v3.QueryExecutorImpl.writeToCopy(QueryExecutorImpl.java:838)
	at org.postgresql.core.v3.CopyInImpl.writeToCopy(CopyInImpl.java:53)
	at org.postgresql.copy.CopyManager.copyIn(CopyManager.java:179)
	at org.postgresql.copy.CopyManager.copyIn(CopyManager.java:161)
	at dk.dmi.appl.evejrfeeder.EvejrDBCopy.run(Unknown Source)
	at java.lang.Thread.run(Thread.java:619)
Sep 15, 2010 6:52:24 PM dk.dmi.appl.evejrfeeder.EvejrDB writeIndexedPointDataSep 15, 2010 6:52:21 PM dk.dmi.appl.evejrfeeder.EvejrDB writeIndexedPointData
INFO: Inserting data in temporary_wind_speed_150_m_3 at valid time 2010-09-15 12:00:00 for 38882 gridpoints



Sep 15, 2010 6:56:47 PM dk.dmi.appl.evejrfeeder.EvejrDB writeIndexedPointData
INFO: Inserting data in temporary_relative_humidity_3 at valid time 2010-09-15 12:00:00 for 38882 gridpoints
Sep 15, 2010 6:56:47 PM dk.dmi.appl.evejrfeeder.EvejrDBCopy run
INFO: ERROR: invalid input syntax for type timestamp: "2010-09-"
   Where: COPY temporary_relative_humidity_3, line 173, column valid_at: "2010-09-"
Sep 15, 2010 6:56:47 PM dk.dmi.appl.evejrfeeder.EvejrDBCopy run
INFO: ERROR: invalid input syntax for type timestamp: "2010-09-"
   Where: COPY temporary_relative_humidity_3, line 173, column valid_at: "2010-09-"
org.postgresql.util.PSQLException: ERROR: invalid input syntax for type timestamp: "2010-09-"
   Where: COPY temporary_relative_humidity_3, line 173, column valid_at: "2010-09-"
	at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2062)
	at org.postgresql.core.v3.QueryExecutorImpl.processCopyResults(QueryExecutorImpl.java:929)
	at org.postgresql.core.v3.QueryExecutorImpl.writeToCopy(QueryExecutorImpl.java:838)
	at org.postgresql.core.v3.CopyInImpl.writeToCopy(CopyInImpl.java:53)
	at org.postgresql.copy.CopyManager.copyIn(CopyManager.java:179)
	at org.postgresql.copy.CopyManager.copyIn(CopyManager.java:161)
	at dk.dmi.appl.evejrfeeder.EvejrDBCopy.run(Unknown Source)
	at java.lang.Thread.run(Thread.java:619)
Sep 15, 2010 6:56:51 PM dk.dmi.appl.evejrfeeder.EvejrDB writeIndexedPointData







On 2010-09-14 09:14, Kim Bisgaard wrote:
> Thanks everybody!
>
> Yes it was tried with the 701 build (I am almost sure that we also 
> tried with cvs-head as of 2 weeks back - will check up on that)
>
> We will correct the connection.close() problem - thanks for pointing 
> that out!
>
> And probably (if we still get problems) try the new build 800 driver.
>
> Thanks again,
> Kim
>
>
> On 2010-09-13 19:27, Samuel Gendler wrote:
>> Your error sure looks exactly like the problem that existed when 
>> using a Reader instead of InputStream with the older driver (701), 
>> though your description implies a different problem.
>>
>> On Mon, Sep 13, 2010 at 7:17 AM, Kris Jurka <books(at)ejurka(dot)com 
>> <mailto:books(at)ejurka(dot)com>> wrote:
>>
>>
>>
>>     On Mon, 13 Sep 2010, Kim Bisgaard wrote:
>>
>>         We are occasionally experiencing problems with our JDBC
>>         implementation of COPY IN.
>>
>>         We have cut our program down to the attached example.
>>
>>
>>     Do you get the failure you've shown from this example?  I get
>>     another failure from the connection being close before the last
>>     copy thread can complete its write operation.  This seems like an
>>     expected failure, but I can't reproduce yours.
>>
>>     org.postgresql.util.PSQLException: Database connection failed
>>     when canceling copy operation
>>            at
>>     org.postgresql.core.v3.QueryExecutorImpl.cancelCopy(QueryExecutorImpl.java:796)
>>            at
>>     org.postgresql.core.v3.CopyOperationImpl.cancelCopy(CopyOperationImpl.java:32)
>>            at
>>     org.postgresql.copy.CopyManager.copyIn(CopyManager.java:150)
>>            at
>>     org.postgresql.copy.CopyManager.copyIn(CopyManager.java:126)
>>            at EvejrDBCopy.run(EvejrDBCopy.java:34)
>>
>>
>>         We use PostgreSQL 8.4, and JDBC 8.4-702.
>>
>>
>>     Your stacktrace shows that you are using 8.4-701, not 702.  702
>>     had some fixes to the copy code when using a Reader, but your
>>     code shows you are using an InputStream, so those changes
>>     shouldn't be relevent.
>>
>>     Kris Jurka
>>
>>     -- 
>>     Sent via pgsql-jdbc mailing list (pgsql-jdbc(at)postgresql(dot)org
>>     <mailto:pgsql-jdbc(at)postgresql(dot)org>)
>>     To make changes to your subscription:
>>     http://www.postgresql.org/mailpref/pgsql-jdbc
>>
>>

Attachment: EvejrDBCopy.java
Description: text/x-java (3.4 KB)

In response to

pgsql-jdbc by date

Next:From: adminDate: 2010-09-17 20:41:40
Subject: Broken pipe error
Previous:From: Craig RingerDate: 2010-09-15 14:41:51
Subject: Re: Upload latest JDBC driver releases to Maven Central

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group