Skip site navigation (1) Skip section navigation (2)

Re: OutOfMemory when inserting stream of unknown length

From: Oliver Jowett <oliver(at)opencloud(dot)com>
To: "Mikko T(dot)" <mtiihone(at)cc(dot)hut(dot)fi>
Cc: pgsql-jdbc(at)postgresql(dot)org
Subject: Re: OutOfMemory when inserting stream of unknown length
Date: 2004-08-20 10:04:40
Message-ID: 4125CCB8.10801@opencloud.com (view raw or flat)
Thread:
Lists: pgsql-jdbc
Mikko T. wrote:

> I think that commiting to send very large data values that are not even 
> guaranteed to exist (or be available) makes it impossible to get the 
> protocol back to known state if for some reason the data cannot be sent 
> or the last command should be aborted. In the current solution the only 
> option seems to be to kill the whole connection but I find that quite an 
> extreme thing to do.

I discussed some alternatives with Kris a while ago but no 
implementation has happened yet. The most promising approach was to pad 
out the parameter in the Bind message, then immediately Close the 
resulting portal without executing it and return an error to the 
application.

> In my opinion the protocol level should support sending the data in 
> chunks (4K - 16K could be optimal). This way the jdbc driver only has to 
> buffer one chunk ahead and not the whole stream and it always knows 
> there won't be any IOExceptions or EOF occuring during the sending. 
> Supporting chunked sending would also allow the jdbc driver to support 
> cancel/rollback the streaming if it needs to or is asked to do so by the 
> database even if there would still be megabytes of data to send.

Right -- something like HTTP/1.1's chunked transfer scheme. We would 
have to have some "partially bound portal" state floating around on the 
server side which seems ugly (and coordinating multiple parameters is 
going to get nasty). Is it worth it? I have a feeling that pgsql-hackers 
may take some convincing; sending data of unknown length doesn't seem 
like it would be widely used, and it'd be quite a lot of complexity to add.

There's nothing much we can do about this with the current protocol 
unfortunately :(

-O

In response to

pgsql-jdbc by date

Next:From: Tobias ZielkeDate: 2004-08-20 14:20:34
Subject: Columns out of range-error
Previous:From: Oliver JowettDate: 2004-08-20 09:59:11
Subject: Re: OutOfMemory when inserting stream of unknown length

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group