Skip site navigation (1) Skip section navigation (2)

Re: Parallel query execution

From: Bruce Momjian <bruce(at)momjian(dot)us>
To: Michael Paquier <michael(dot)paquier(at)gmail(dot)com>
Cc: PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Parallel query execution
Date: 2013-01-16 02:29:01
Message-ID: 20130116022901.GA369@momjian.us (view raw or flat)
Thread:
Lists: pgsql-hackers
On Wed, Jan 16, 2013 at 09:11:20AM +0900, Michael Paquier wrote:
> 
> 
> On Wed, Jan 16, 2013 at 7:14 AM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> 
>     I mentioned last year that I wanted to start working on parallelism:
> 
>             https://wiki.postgresql.org/wiki/Parallel_Query_Execution
> 
>     Years ago I added thread-safety to libpq.  Recently I added two parallel
>     execution paths to pg_upgrade.  The first parallel path allows execution
>     of external binaries pg_dump and psql (to restore).  The second parallel
>     path does copy/link by calling fork/thread-safe C functions.  I was able
>     to do each in 2-3 days.
> 
>     I believe it is time to start adding parallel execution to the backend.
>     We already have some parallelism in the backend:
>     effective_io_concurrency and helper processes.  I think it is time we
>     start to consider additional options.
> 
>     Parallelism isn't going to help all queries, in fact it might be just a
>     small subset, but it will be the larger queries.  The pg_upgrade
>     parallelism only helps clusters with multiple databases or tablespaces,
>     but the improvements are significant.
> 
>     I have summarized my ideas by updating our Parallel Query Execution wiki
>     page:
> 
>             https://wiki.postgresql.org/wiki/Parallel_Query_Execution
> 
>     Please consider updating the page yourself or posting your ideas to this
>     thread.  Thanks.
> 
> Honestly that would be a great feature, and I would be happy helping working on
> it.
> Taking advantage of parallelism in a server with multiple core, especially for
> things like large sorting operations would be great.
> Just thinking loudly, but wouldn't it be the role of the planner to determine
> if such or such query is worth using parallelism? The executor would then be in
> charge of actually firing the tasks in parallel that planner has determined
> necessary to do.

Yes, it would probably be driven off of the optimizer statistics.

-- 
  Bruce Momjian  <bruce(at)momjian(dot)us>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +


In response to

pgsql-hackers by date

Next:From: Claudio FreireDate: 2013-01-16 02:36:55
Subject: Re: Parallel query execution
Previous:From: Tom LaneDate: 2013-01-16 01:50:04
Subject: Re: Curious buildfarm failures (fwd)

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group