Re: pg_dump / copy bugs with "big lines" ?

I wrote:

> postgres=# \copy big2 to /dev/null
> lost synchronization with server: got message type "d", length -1568669676
>
> The backend aborts with the following backtrace:
>
> Program terminated with signal 6, Aborted.
> #0 0x00007f82ee68e165 in raise () from /lib/x86_64-linux-gnu/libc.so.6
> #1 0x00007f82ee6913e0 in abort () from /lib/x86_64-linux-gnu/libc.so.6
> #2 0x00007f82ee6c839b in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> #3 0x00007f82ee6d1be6 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> #4 0x00007f82ee6d698c in free () from /lib/x86_64-linux-gnu/libc.so.6
> #5 0x00000000007b5a89 in AllocSetDelete (context=0xffffffffffffffff)
> at aset.c:643
> #6 0x00000000007b63e3 in MemoryContextDelete (context=0x1fa58c8) at
> mcxt.c:229
> #7 0x000000000055fa25 in CopyTo (cstate=0x1fb1050) at copy.c:1967

The cause of the crash turns out to be, in enlargeStringInfo():

needed += str->len + 1; /* total space required now */

needed is an int and str->len is an int64, so it overflows when the
size has to grow beyond 2^31 bytes, fails to enlarge the buffer and
overwrites memory after it.

When fixing it with a local int64 copy of the variable, the backend
no longer crashes and COPY big2 TO 'file' appears to work.

However, getting it to the client with \copy big2 to 'file'
still produces the error in psql:
lost synchronization with server: got message type "d"
and leaves an empty file, so there are more problems to solve to
go beyond 2GB text per row.

Or maybe another approach would be to advertise that this is the maximum
for a row in text mode, and limit the backend's string buffer to this size
for the time being? Notwithstanding that there's still the other direction
client->server to test.

Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite