Re: extending relations more efficiently

From: Stephen Frost <sfrost(at)snowman(dot)net>
To: Robert Haas <robertmhaas(at)gmail(dot)com>
Cc: Andres Freund <andres(at)anarazel(dot)de>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: extending relations more efficiently
Date: 2012-05-01 19:48:04
Message-ID: 20120501194804.GS1267@tamriel.snowman.net
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

* Robert Haas (robertmhaas(at)gmail(dot)com) wrote:
> I think the first thing we need here is a good test case, so we're
> clear on what we're trying to solve. I was just hoping to make file
> extension *faster* and what you and Simon are talking about is making
> it scale better in the face of heavy parallelism; obviously it would
> be nice to do both things, but they are different problems. Any old
> bulk-loading test will benefit from a raw performance improvement, but
> to test a scalability improvement we would need some kind of test case
> involving parallel bulk loads, or some other kind of parallel activity
> that causes rapid table growth. That's not something I've frequently
> run into, but I'd be willing to put a bit of time into it if we can
> nail down what we're talking about.

Try loading the TIGER 2011 data into a single table, where you load each
county (or perhaps state) in a separate, parallel, process. That's what
I was doing when I hit this lock full-force and bitched about it on this
list.

I'd be happy to help construct that case, as well as test any changes in
this area which might help address it (on a 10G SSD-backed SAN..).

Thanks,

Stephen

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Alvaro Herrera 2012-05-01 19:48:56 Re: port _srv.o makefile rules don't observe dependency tracking
Previous Message Noah Misch 2012-05-01 19:36:22 Re: proposal: additional error fields