Re: pg_dump / copy bugs with "big lines" ?

I wrote:

> If splitting the table into 3 fields, each smaller than 512MB:
>
> postgres=# create table big2 as select
> substring(binarycol from 1 for 300*1024*1024) as b1,
> substring(binarycol from 1+300*1024*1024 for 300*1024*1024) as b2 ,
> substring(binarycol from 1+600*1024*1024 for 400*1024*1024) as b3
> from big;

I've tried adding another large field to see what happens if the whole row
exceeds 2GB, and data goes to the client rather than to a file.

postgres=# alter table big2 add b4 bytea;
postgres=# update big2 set b4=b1;

So big2 has 4 bytea columns with 300+300+400+300 MB on a single row.

Then I'm trying to \copy this from a 9.5.1 backend with patch applied,
configured with --enable-debug, on Debian8 x86-64 with 64GB of RAM
and all defaults in postgresql.conf

My idea was to check if the client side was OK with that much data on
a single COPY row, but it turns out that the server is not OK anyway.

postgres=# \copy big2 to /dev/null
lost synchronization with server: got message type "d", length -1568669676

The backend aborts with the following backtrace:

Program terminated with signal 6, Aborted.
#0 0x00007f82ee68e165 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f82ee6913e0 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007f82ee6c839b in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007f82ee6d1be6 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00007f82ee6d698c in free () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x00000000007b5a89 in AllocSetDelete (context=0xffffffffffffffff)
at aset.c:643
#6 0x00000000007b63e3 in MemoryContextDelete (context=0x1fa58c8) at
mcxt.c:229
#7 0x000000000055fa25 in CopyTo (cstate=0x1fb1050) at copy.c:1967
#8 DoCopyTo (cstate=cstate(at)entry=0x1fb1050) at copy.c:1778
#9 0x0000000000562ea9 in DoCopy (stmt=stmt(at)entry=0x1f796d0,
queryString=queryString(at)entry=0x1f78c60 "COPY big2 TO STDOUT ",
processed=processed(at)entry=0x7fff22103338) at copy.c:961
#10 0x00000000006b4440 in standard_ProcessUtility (parsetree=0x1f796d0,
queryString=0x1f78c60 "COPY big2 TO STDOUT ", context=<optimized out>,
params=0x0, dest=0x1f79a30, completionTag=0x7fff22103680 "")
at utility.c:541
#11 0x00000000006b1937 in PortalRunUtility (portal=0x1f26680,
utilityStmt=0x1f796d0, isTopLevel=1 '\001', dest=0x1f79a30,
completionTag=0x7fff22103680 "") at pquery.c:1183
#12 0x00000000006b2535 in PortalRunMulti (portal=portal(at)entry=0x1f26680,
isTopLevel=isTopLevel(at)entry=1 '\001', dest=dest(at)entry=0x1f79a30,
altdest=altdest(at)entry=0x1f79a30,
completionTag=completionTag(at)entry=0x7fff22103680 "") at pquery.c:1314
#13 0x00000000006b3022 in PortalRun (portal=portal(at)entry=0x1f26680,
count=count(at)entry=9223372036854775807,
isTopLevel=isTopLevel(at)entry=1 '\001', dest=dest(at)entry=0x1f79a30,
altdest=altdest(at)entry=0x1f79a30,
completionTag=completionTag(at)entry=0x7fff22103680 "") at pquery.c:812
#14 0x00000000006b0393 in exec_simple_query (
query_string=0x1f78c60 "COPY big2 TO STDOUT ") at postgres.c:1104
#15 PostgresMain (argc=<optimized out>, argv=argv(at)entry=0x1f0d240,
dbname=0x1f0d0f0 "postgres", username=<optimized out>) at postgres.c:4030
#16 0x000000000047072b in BackendRun (port=0x1f2d230) at postmaster.c:4239
#17 BackendStartup (port=0x1f2d230) at postmaster.c:3913
#18 ServerLoop () at postmaster.c:1684
#19 0x000000000065b8cd in PostmasterMain (argc=argc(at)entry=3,
argv=argv(at)entry=0x1f0c330) at postmaster.c:1292
#20 0x0000000000471161 in main (argc=3, argv=0x1f0c330) at main.c:223

The server log has this:

STATEMENT: COPY big2 TO STDOUT
*** glibc detected *** postgres: postgres postgres [local] COPY: double free
or corruption (out): 0x00007f8234929010 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x75be6)[0x7f82ee6d1be6]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x6c)[0x7f82ee6d698c]
postgres: postgres postgres [local] COPY[0x7b5a89]
postgres: postgres postgres [local] COPY(MemoryContextDelete+0x43)[0x7b63e3]
postgres: postgres postgres [local] COPY[0x55fa25]
postgres: postgres postgres [local] COPY(DoCopy+0x479)[0x562ea9]
postgres: postgres postgres [local]
COPY(standard_ProcessUtility+0x590)[0x6b4440]
postgres: postgres postgres [local] COPY[0x6b1937]
postgres: postgres postgres [local] COPY[0x6b2535]
postgres: postgres postgres [local] COPY(PortalRun+0x202)[0x6b3022]
postgres: postgres postgres [local] COPY(PostgresMain+0x1493)[0x6b0393]
postgres: postgres postgres [local] COPY[0x47072b]
postgres: postgres postgres [local] COPY(PostmasterMain+0xe7d)[0x65b8cd]
postgres: postgres postgres [local] COPY(main+0x3e1)[0x471161]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd)[0x7f82ee67aead]
postgres: postgres postgres [local] COPY[0x4711c9]

I will try other tests without bytea exported in text format but with
several huge text columns.

Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite