Skip site navigation (1) Skip section navigation (2)

Re: Best COPY Performance

From: "Worky Workerson" <worky(dot)workerson(at)gmail(dot)com>
To: "Craig A(dot) James" <cjames(at)modgraph-usa(dot)com>
Cc: "Jim C(dot) Nasby" <jim(at)nasby(dot)net>, "Merlin Moncure" <mmoncure(at)gmail(dot)com>, pgsql-performance(at)postgresql(dot)org
Subject: Re: Best COPY Performance
Date: 2006-10-25 12:03:38
Message-ID: ce4072df0610250503g26f32cb7r4e668613c8665774@mail.gmail.com (view raw or flat)
Thread:
Lists: pgsql-performance
On 10/25/06, Craig A. James <cjames(at)modgraph-usa(dot)com> wrote:
> Jim C. Nasby wrote:
> > Well, given that perl is using an entire CPU, it sounds like you should
> > start looking either at ways to remove some of the overhead from perl,
> > or to split that perl into multiple processes.
>
> I use Perl for big database copies (usually with some processing/transformation along the
> way) and I've never seen 100% CPU usage except for brief periods, even when copying
> BLOBS and such.  My typical copy divides operations into blocks, for example doing

I'm just doing CSV style transformations (and calling a lot of
functions along the way), but the end result is a straight bulk load
of data into a blank database.  And we've established that Postgres
can do *way* better than what I am seeing, so its not suprising that
perl is using 100% of a CPU.

However, I am still curious as to the rather slow COPYs from psql to
local disks.  Like I mentioned previously, I was only seeing about 5.7
MB/s (1.8 GB / 330 seconds), where it seemed like others were doing
substantially better.  What sorts of things should I look into?

Thanks!

In response to

Responses

pgsql-performance by date

Next:From: Alex StapletonDate: 2006-10-25 12:16:15
Subject: Re: Best COPY Performance
Previous:From: Craig A. JamesDate: 2006-10-25 05:36:04
Subject: Re: Best COPY Performance

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group