Re: OutOfMemory when inserting stream of unknown length

Mikko T. wrote:

> I think that commiting to send very large data values that are not even
> guaranteed to exist (or be available) makes it impossible to get the
> protocol back to known state if for some reason the data cannot be sent
> or the last command should be aborted. In the current solution the only
> option seems to be to kill the whole connection but I find that quite an
> extreme thing to do.

I discussed some alternatives with Kris a while ago but no
implementation has happened yet. The most promising approach was to pad
out the parameter in the Bind message, then immediately Close the
resulting portal without executing it and return an error to the
application.

> In my opinion the protocol level should support sending the data in
> chunks (4K - 16K could be optimal). This way the jdbc driver only has to
> buffer one chunk ahead and not the whole stream and it always knows
> there won't be any IOExceptions or EOF occuring during the sending.
> Supporting chunked sending would also allow the jdbc driver to support
> cancel/rollback the streaming if it needs to or is asked to do so by the
> database even if there would still be megabytes of data to send.

Right -- something like HTTP/1.1's chunked transfer scheme. We would
have to have some "partially bound portal" state floating around on the
server side which seems ugly (and coordinating multiple parameters is
going to get nasty). Is it worth it? I have a feeling that pgsql-hackers
may take some convincing; sending data of unknown length doesn't seem
like it would be widely used, and it'd be quite a lot of complexity to add.

There's nothing much we can do about this with the current protocol
unfortunately :(

-O