From: | Stefan Kaltenbrunner <stefan(at)kaltenbrunner(dot)cc> |
---|---|
To: | Jeff <threshar(at)real(dot)jefftrout(dot)com> |
Cc: | Peter Eisentraut <peter_e(at)gmx(dot)net>, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, Josh Berkus <josh(at)agliodbs(dot)com>, Joachim Wieland <joe(at)mcknight(dot)de>, pgsql-hackers(at)postgresql(dot)org |
Subject: | Re: Parallel pg_dump for 9.1 |
Date: | 2010-04-01 13:50:37 |
Message-ID: | 4BB4A4AD.6080001@kaltenbrunner.cc |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-hackers |
Jeff wrote:
>
> On Mar 30, 2010, at 8:15 AM, Stefan Kaltenbrunner wrote:
>
>> Peter Eisentraut wrote:
>>> On tis, 2010-03-30 at 08:39 +0200, Stefan Kaltenbrunner wrote:
>>>> on fast systems pg_dump is completely CPU bottlenecked
>>> Might be useful to profile why that is. I don't think pg_dump has
>>> historically been developed with CPU efficiency in mind.
>>
>> It's not pg_dump that is the problem - it is COPY that is the limit.
>> In my specific case als the fact that a lot of the columns are bytea
>> adds to the horrible CPU overhead (fixed in 9.0). Still our bulk load
>> & unload performance is still way slower on a per core comparision
>> than a lot of other databases :(
>>
>
> Don't forget the zlib compression used in -Fc (unless you use -Z0) takes
> a fair amount of cpu too.
> I did some tests and it turned out that -Z0 actually took longer than
> -Z1 simply because there was a lot more data to write out, thus I became
> IO bound not CPU bound.
>
> There's a thing called pigz around that is a parallel gzip
> implementation - wonder how much of that could be adapted to pg_dumps
> use as compression does use a considerable amount of time (even at
> -Z1). The biggest problem I can immediately see is that it uses threads.
all my numbers are with -Z0 and it is the backend (COPY and/or index
creation) that is the limit. If you start using compression you are
shifting the load to pg_dump.
Stefan
From | Date | Subject | |
---|---|---|---|
Next Message | Bruce Momjian | 2010-04-01 13:51:59 | Re: pgsql: Add contraint exclusion section to contraint docs. |
Previous Message | Greg Sabino Mullane | 2010-04-01 13:43:15 | Re: missing schema qualifications in psql |