Re: bytea size limit?

Michael Privat wrote:
> Here you go. Coded against the CVS HEAD. Let me know if you want me to
> change anything. Obviously, because I have to send the size ahead,
> there is no other way but store the entire byte[] in memory (can't
> stream), but with this code, I was able to load a blob of 30Mb+ (with
> a heap size of 32Mb) against 1.5Mb yesterday with the same heap. So
> there is some improvement.
>
> http://jupiter.incodea.com/trans/pgsql-jdbc.diff

Comments:

Does this work for V2 connections? I didn't see a patch to the V2
execution path, but you've changed how setBytes() binds its argument.

The argument to setBytes() is not copied when bound; changes made to the
array after the setBytes() call but before query execution will be
visible. Personally I don't mind this behaviour but others disagree :)
-- see earlier discussion about mutable parameters in the pgsql-jdbc
archives.

Individually escaping and encoding each byte (in SendBytea and
GetByteaSize) seems overly expensive. We should be able to do this in
blocks at a minimum.

Why is EncodedStream necessary? Can't you just unroll that logic
directly in SendBytea()?

It's probably possible to work around needing the whole array in memory
(i.e. we could stream from an InputStream + length via setBinaryStream).
If we escape *every* byte, we need exactly 5 bytes of UTF8 output for
every input byte ('\' and '0'-'9' are all single bytes in UTF8, I
believe) so we could compute the output size directly from the input length.

====

That said, this patch is a lot simpler than I expected it to be! It
looks like a good solution until the driver knows how to speak V3
fluently. If you like, I'll see if I can rework your patch over the next
couple of days to address the above points.

-O