Skip site navigation (1) Skip section navigation (2)

Re: pg_dump additional options for performance

From: Gregory Stark <stark(at)enterprisedb(dot)com>
To: "Tom Lane" <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: "Simon Riggs" <simon(at)2ndquadrant(dot)com>, "Jeff Davis" <pgsql(at)j-davis(dot)com>, <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: pg_dump additional options for performance
Date: 2008-02-26 20:14:49
Message-ID: 874pbvfpae.fsf@oxford.xeocode.com (view raw or flat)
Thread:
Lists: pgsql-hackers
"Tom Lane" <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:

> Simon Riggs <simon(at)2ndquadrant(dot)com> writes:
>> I've not been advocating improving pg_restore, which is where the -Fc
>> tricks come in.
>> ...
>> I see you thought I meant pg_restore. I don't thinking extending
>> pg_restore in that way is of sufficiently generic use to make it
>> worthwhile. Extending psql would be worth it, since not all psql scripts
>> come from pg_dump.
>
> OK, the reason I didn't grasp what you are proposing is that it's insane.
>
> We can easily, and backwards-compatibly, improve pg_restore to do
> concurrent restores.  Trying to make psql do something like this will
> require a complete rewrite, and there is no prospect that it will work
> for any input that didn't come from (an updated version of) pg_dump
> anyway.  

I didn't read everything in the thread previously so I'm not sure if this is
what Simon had in mind. But I think one thing that could be done in parallel
even in psql scripts is index builds. That doesn't help speed up COPY but it
does speed up a case where we currently are limited by only occupying a single
cpu. And I would expect it to play well With synchronized scans too.

The "complete rewrite" in this case would be the "concurrent psql" patch I
submitted a while back. I think it's a bit of a mess right now because I was
trying to chase down some bugs with sigint handling so I've been thinking of
rewriting it.

I think this is a low-hanging fruit which would help a lot of users. The
ability to load multiple COPY dumps would be the other piece of the puzzle but
personally I'm not sure how to tackle that.

-- 
  Gregory Stark
  EnterpriseDB          http://www.enterprisedb.com
  Ask me about EnterpriseDB's PostGIS support!

In response to

Responses

pgsql-hackers by date

Next:From: Gregory StarkDate: 2008-02-26 20:22:01
Subject: Re: Proposed changes to DTrace probe implementation
Previous:From: Neil ConwayDate: 2008-02-26 20:06:39
Subject: Re: code cleanup of timestamp code

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group