Skip site navigation (1) Skip section navigation (2)

Re: Improve COPY performance for large data sets

From: Alan Hodgson <ahodgson(at)simkin(dot)ca>
To: pgsql-performance(at)postgresql(dot)org
Subject: Re: Improve COPY performance for large data sets
Date: 2008-09-10 17:14:34
Message-ID: 200809101014.34249@hal.medialogik.com (view raw or flat)
Thread:
Lists: pgsql-performance
On Wednesday 10 September 2008, Ryan Hansen <ryan(dot)hansen(at)brightbuilders(dot)com> 
wrote:
>Currently it's taking about 12 hours to complete on a 64 bit
> server with 3 GB memory allocated (shared_buffer), single SATA 320 GB
> drive.  I don't seem to get any improvement running the same operation
> on a dual opteron dual-core, 16 GB server.
>
> I'm not asking for someone to solve my problem, just some direction in
> the best ways to tune for faster bulk loading, since this will be a
> fairly regular operation for our application (assuming it can work this
> way).  I've toyed with the maintenance_work_mem and some of the other
> params, but it's still way slower than it seems like it should be.
> So any contributions are much appreciated.

Your drive subsystem, such as it is, is inappropriate for a database. Your 
bottleneck is your drive. 

Turning fsync off might help. You should also drop all indexes on the table 
before the COPY and add them back after (which would eliminate a lot of 
random I/O during the COPY).

-- 
Alan

In response to

Responses

pgsql-performance by date

Next:From: Bill MoranDate: 2008-09-10 17:16:06
Subject: Re: Improve COPY performance for large data sets
Previous:From: Ryan HansenDate: 2008-09-10 17:14:23
Subject: Re: Improve COPY performance for large data sets

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group