Skip site navigation (1) Skip section navigation (2)

Re: pg_dump and thousands of schemas

From: Stephen Frost <sfrost(at)snowman(dot)net>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: Tatsuo Ishii <ishii(at)postgresql(dot)org>, pgsql-performance(at)postgresql(dot)org,pgsql-hackers(at)postgresql(dot)org
Subject: Re: pg_dump and thousands of schemas
Date: 2012-05-31 04:01:49
Message-ID: 20120531040149.GC1267@tamriel.snowman.net (view raw or flat)
Thread:
Lists: pgsql-hackerspgsql-performance
* Tom Lane (tgl(at)sss(dot)pgh(dot)pa(dot)us) wrote:
> Tatsuo Ishii <ishii(at)postgresql(dot)org> writes:
> > Shall I commit to master and all supported branches?
> 
> I'm not excited by this patch.  It dodges the O(N^2) lock behavior for
> the initial phase of acquiring the locks, but it does nothing for the
> lock-related slowdown occurring in all pg_dump's subsequent commands.
> I think we really need to get in the server-side fix that Jeff Janes is
> working on, and then re-measure to see if something like this is still
> worth the trouble.  I am also a tad concerned about whether we might not
> have problems with parsing memory usage, or some such, with thousands of
> tables being listed in a single command.

I can't imagine a case where it's actually better to incur the latency
penalty (which is apparently on the order of *minutes* of additional
time here..) than to worry about the potential memory usage of having to
parse such a command.

If that's really a concern, where is that threshold, and could we simply
cap pg_dump's operations based on it?  Is 1000 alright?  Doing a 'lock'
w/ 1000 tables at a time is still going to be hugely better than doing
them individually and the amount of gain between every-1000 and
all-at-once is likely to be pretty minimal anyway...

The current situation where the client-to-server latency accounts for
multiple minutes of time is just ridiculous, however, so I feel we need
some form of this patch, even if the server side is magically made much
faster.  The constant back-and-forth isn't cheap.

	Thanks,

		Stephen

In response to

Responses

pgsql-performance by date

Next:From: Tom LaneDate: 2012-05-31 04:18:17
Subject: Re: pg_dump and thousands of schemas
Previous:From: Tom LaneDate: 2012-05-31 03:54:30
Subject: Re: pg_dump and thousands of schemas

pgsql-hackers by date

Next:From: Tom LaneDate: 2012-05-31 04:18:17
Subject: Re: pg_dump and thousands of schemas
Previous:From: Tom LaneDate: 2012-05-31 03:54:30
Subject: Re: pg_dump and thousands of schemas

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group