Skip site navigation (1) Skip section navigation (2)

Re: Best COPY Performance

From: Stefan Kaltenbrunner <stefan(at)kaltenbrunner(dot)cc>
To: Luke Lonergan <llonergan(at)greenplum(dot)com>
Cc: "Spiegelberg, Greg" <gspiegelberg(at)cranel(dot)com>, Worky Workerson <worky(dot)workerson(at)gmail(dot)com>, Merlin Moncure <mmoncure(at)gmail(dot)com>, pgsql-performance(at)postgresql(dot)org
Subject: Re: Best COPY Performance
Date: 2006-10-30 15:57:19
Message-ID: 454620DF.1000003@kaltenbrunner.cc (view raw or flat)
Thread:
Lists: pgsql-performance
Luke Lonergan wrote:
> Greg,
> 
> On 10/30/06 7:09 AM, "Spiegelberg, Greg" <gspiegelberg(at)cranel(dot)com> wrote:
> 
>> I broke that file into 2 files each of 550K rows and performed 2
>> simultaneous COPY's after dropping the table, recreating, issuing a sync
>> on the system to be sure, &c and nearly every time both COPY's finish in
>> 12 seconds.  About a 20% gain to ~91K rows/second.
>>
>> Admittedly, this was a pretty rough test but a 20% savings, if it can be
>> put into production, is worth exploring for us.
> 
> Did you see whether you were I/O or CPU bound in your single threaded COPY?
> A 10 second "vmstat 1" snapshot would tell you/us.
> 
> With Mr. Workerson (:-) I'm thinking his benefit might be a lot better
> because the bottleneck is the CPU and it *may* be the time spent in the
> index building bits.
> 
> We've found that there is an ultimate bottleneck at about 12-14MB/s despite
> having sequential write to disk speeds of 100s of MB/s.  I forget what the
> latest bottleneck was.

I have personally managed to load a bit less then 400k/s (5 int columns 
no indexes) - on very fast disk hardware - at that point postgresql is 
completely CPU bottlenecked (2,6Ghz Opteron).
Using multiple processes to load the data will help to scale up to about 
  900k/s (4 processes on 4 cores).


Stefan

In response to

Responses

pgsql-performance by date

Next:From: Stefan KaltenbrunnerDate: 2006-10-30 16:23:44
Subject: Re: Best COPY Performance
Previous:From: Steinar H. GundersonDate: 2006-10-30 15:49:57
Subject: Re: Strange plan in pg 8.1.0

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group