Re: Performance while loading data and indexing

From: "Shridhar Daithankar" <shridhar_daithankar(at)persistent(dot)co(dot)in>
To: paolo(dot)cassago(at)talentmanager(dot)com
Cc: pgsql-hackers(at)postgresql(dot)org
Subject: Re: Performance while loading data and indexing
Date: 2002-09-26 09:13:20
Message-ID: 3D931D08.1695.135D474B@localhost
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general pgsql-hackers pgsql-performance

On 26 Sep 2002 at 10:51, paolo(dot)cassago(at)talentmanager(dot)c wrote:

> Hi,
> it seems you have to cluster it, I don't think you have another choise.

Hmm.. That didn't occur to me...I guess some real time clustering like usogres
would do. Unless it turns out to be a performance hog..

But this is just insert and select. No updates no deletes(Unless customer makes
a 180 degree turn) So I doubt if clustering will help. At the most I can
replicate data across machines and spread queries on them. Replication overhead
as a down side and low query load on each machine as upside..

> I'm retrieving the configuration of our postgres servers (I'm out of office
> now), so I can send it to you. I was quite disperate about performance, and
> I was thinking to migrate the data on an oracle database. Then I found this
> configuration on the net, and I had a succesfull increase of performance.

In this case, we are upto postgresql because we/our customer wants to keep the
costs down..:-) Even they are asking now if it's possible to keep hardware
costs down as well. That's getting some funny responses here but I digress..

> Maybe this can help you.
>
> Why you use copy to insert records? I usually use perl scripts, and they
> work well .

Performance reasons. As I said in one of my posts earlier, putting upto 100K
records in one transaction in steps of 10K did not reach performance of copy.
As Tom said rightly, it was a 4-1 ratio despite using transactions..

Thanks once again..
Bye
Shridhar

--
Secretary's Revenge: Filing almost everything under "the".

In response to

Browse pgsql-general by date

  From Date Subject
Next Message Justin Clift 2002-09-26 09:17:32 Re: Performance while loading data and indexing
Previous Message Martijn van Oosterhout 2002-09-26 09:05:19 Re: Performance while loading data and indexing

Browse pgsql-hackers by date

  From Date Subject
Next Message Justin Clift 2002-09-26 09:17:32 Re: Performance while loading data and indexing
Previous Message Martijn van Oosterhout 2002-09-26 09:05:19 Re: Performance while loading data and indexing

Browse pgsql-performance by date

  From Date Subject
Next Message Justin Clift 2002-09-26 09:17:32 Re: Performance while loading data and indexing
Previous Message Martijn van Oosterhout 2002-09-26 09:05:19 Re: Performance while loading data and indexing