Re: Parallel Append implementation

From: Andres Freund <andres(at)anarazel(dot)de>
To: Robert Haas <robertmhaas(at)gmail(dot)com>
Cc: Amit Khandekar <amitdkhan(dot)pg(at)gmail(dot)com>, Ashutosh Bapat <ashutosh(dot)bapat(at)enterprisedb(dot)com>, pgsql-hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Parallel Append implementation
Date: 2017-04-04 20:13:48
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On 2017-04-04 08:01:32 -0400, Robert Haas wrote:
> On Tue, Apr 4, 2017 at 12:47 AM, Andres Freund <andres(at)anarazel(dot)de> wrote:
> > I don't think the parallel seqscan is comparable in complexity with the
> > parallel append case. Each worker there does the same kind of work, and
> > if one of them is behind, it'll just do less. But correct sizing will
> > be more important with parallel-append, because with non-partial
> > subplans the work is absolutely *not* uniform.
> Sure, that's a problem, but I think it's still absolutely necessary to
> ramp up the maximum "effort" (in terms of number of workers)
> logarithmically. If you just do it by costing, the winning number of
> workers will always be the largest number that we think we'll be able
> to put to use - e.g. with 100 branches of relatively equal cost we'll
> pick 100 workers. That's not remotely sane.

I'm quite unconvinced that just throwing a log() in there is the best
way to combat that. Modeling the issue of starting more workers through
tuple transfer, locking, startup overhead costing seems a better to me.

If the goal is to compute the results of the query as fast as possible,
and to not use more than max_parallel_per_XXX, and it's actually
beneficial to use more workers, then we should. Because otherwise you
really can't use the resources available.

- Andres

In response to


Browse pgsql-hackers by date

  From Date Subject
Next Message Andres Freund 2017-04-04 20:19:15 Re: Logical decoding on standby
Previous Message Fabien COELHO 2017-04-04 20:05:10 psql - add special variable to reflect the last query status