Skip site navigation (1) Skip section navigation (2)

out of memory during COPY .. FROM

From: Tom Lanyon <tom+pgsql-hackers(at)oneshoeco(dot)com>
To: pgsql-hackers(at)postgresql(dot)org
Subject: out of memory during COPY .. FROM
Date: 2011-02-01 10:32:47
Message-ID: 906B0BFE-F2CF-45E4-86C8-46C328C6F104@oneshoeco.com (view raw or flat)
Thread:
Lists: pgsql-hackers
List,

Can anyone suggest where the below error comes from, given I'm attempting to load HTTP access log data with reasonably small row and column value lengths?

	logs=# COPY raw FROM '/path/to/big/log/file' DELIMITER E'\t' CSV;
	ERROR:  out of memory
	DETAIL:  Cannot enlarge string buffer containing 1073712650 bytes by 65536 more bytes.
	CONTEXT:  COPY raw, line 613338983

It was suggested in #postgresql that I'm reaching the 1GB MaxAllocSize - but I would have thought this would only be a constraint against either large values for specific columns or for whole rows. It's worth noting that this is after 613 million rows have already been loaded (somewhere around 100GB of data) and that I'm running this COPY after the "CREATE TABLE raw ..." in a single transaction.

I've looked at line 613338983 in the file being loaded (+/- 10 rows) and can't see anything out of the ordinary.

Disclaimer: I know nothing of PostgreSQL's internals, please be gentle!

Regards,
Tom


Responses

pgsql-hackers by date

Next:From: Thom BrownDate: 2011-02-01 10:35:11
Subject: Re: [HACKERS] Add reference to client_encoding parameter
Previous:From: Nicolas BarbierDate: 2011-02-01 10:05:17
Subject: Re: bad links in messages from commits

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group