Skip site navigation (1) Skip section navigation (2)

Re: multi-worker pg_restore was: 8.3 / 8.2.6 restore comparison

From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: "Joshua D(dot) Drake" <jd(at)commandprompt(dot)com>
Cc: Jeff Davis <pgsql(at)j-davis(dot)com>, Heikki Linnakangas <heikki(at)enterprisedb(dot)com>, Stefan Kaltenbrunner <stefan(at)kaltenbrunner(dot)cc>, Luke Lonergan <llonergan(at)greenplum(dot)com>, Greg Smith <gsmith(at)gregsmith(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: multi-worker pg_restore was: 8.3 / 8.2.6 restore comparison
Date: 2008-02-27 00:03:57
Message-ID: 10396.1204070637@sss.pgh.pa.us (view raw or flat)
Thread:
Lists: pgsql-hackers
"Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
> Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> How exactly are you allocating tasks to threads in this prototype,
>> anyway?

> Right there is no balance here. Let me explain what I did. I performed
> a pg_restore -l to get the TOC file. I then broke that file up into
> five other files.

> prefix = schema (minus indexes, constraints)
> data = data
> pk = primary keys 
> index = indexes
> triggers_constraints = well triggers and cosntraints (foreign keys in
> this instance)

> The first step of the script loads prefix. It then splits the data
> file into -n- number of files and launches -n- number of
> pg_restore processes with -L.

> It runs through all data, then starts on pk in the exact same manner
> and then indxex etc...

So you have four serialization points not just one; at each one the
slowest subtask forces everyone else to wait, even if there's work that
could potentially be done on other tables.  This is fine for a
quick-and-dirty proof of concept but it's certainly not how we'd want to
implement the real thing.  But I doubt you can get much further without
putting some actual dependency awareness into it.

			regards, tom lane

In response to

Responses

pgsql-hackers by date

Next:From: Joshua D. DrakeDate: 2008-02-27 00:24:01
Subject: Re: multi-worker pg_restore was: 8.3 / 8.2.6 restore comparison
Previous:From: Tom LaneDate: 2008-02-26 23:59:16
Subject: Re: Required make version

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group