Skip site navigation (1) Skip section navigation (2)

Re: pg_restore --multi-thread

From: Andrew Dunstan <andrew(at)dunslane(dot)net>
To: jd(at)commandprompt(dot)com
Cc: Peter Eisentraut <peter_e(at)gmx(dot)net>, " Cédric Villema =?ISO-8859-1?Q?in?=" <cedric(dot)villemain(at)dalibo(dot)com>, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: pg_restore --multi-thread
Date: 2009-02-20 17:34:15
Message-ID: 499EE997.40409@dunslane.net (view raw or flat)
Thread:
Lists: pgsql-hackers

Joshua D. Drake wrote:
> On Fri, 2009-02-20 at 09:33 -0500, Andrew Dunstan wrote:
>
>   
>> The short answer is that we don't know yet. There is anecdotal evidence 
>> that the number of CPUs on the server is a good place to start, but we 
>> should be honest enough to say that this is a new feature and we are 
>> still gathering information about its performance.  If you want to give 
>> some advice, then I think the best advice is to try a variety of 
>> settings to see what works best for you, and if you have a good set of 
>> figures report it back to us.
>>     
>
> There has been some fairly heavy testing and research that caused the
> patch in the first place. The thread is here:
>
> http://archives.postgresql.org/pgsql-hackers/2008-02/msg00695.php
>
> It is a long thread. The end was result was the fastest restore time for
> 220G was performed with 24 threads with an 8 core box. It came in at 3.5
> hours.
>
> http://archives.postgresql.org/pgsql-hackers/2008-02/msg01092.php
>
> It is important to point out that this was a machine with 50 spindles.
> Which is where your bottleneck is going to be immediately after solving
> the CPU bound nature of the problem.
>
> So although the CPU question is easily answered, the IO is not. IO is
> extremely variable in its performance.
>
>
>   

Yes, quite true. But parallel restore doesn't work quite the same way 
your original shell scripts did. It tries harder to keep the job pool 
continuously occupied, and so its best number of jobs is likely to be a 
bit lower then yours.

But you are right that there isn't a simple formula.

cheers

andrew

In response to

Responses

pgsql-hackers by date

Next:From: Kenneth MarshallDate: 2009-02-20 17:37:06
Subject: Re: pg_restore --multi-thread
Previous:From: Joshua D. DrakeDate: 2009-02-20 17:22:58
Subject: Re: pg_restore --multi-thread

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group