Re: [WIP] speeding up GIN build with parallel workers

On Thu, Mar 17, 2016 at 2:56 PM, Constantin S. Pan <kvapen(at)gmail(dot)com> wrote:
>
> On Thu, 17 Mar 2016 13:21:32 +0530
> Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
>
> > On Wed, Mar 16, 2016 at 7:50 PM, Constantin S. Pan <kvapen(at)gmail(dot)com>
> > wrote:
> > >
> > > On Wed, 16 Mar 2016 18:08:38 +0530
> > > Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> > >
> > > >
> > > > Why backend just waits, why can't it does the same work as any
> > > > worker does? In general, for other parallelism features the
> > > > backend also behaves the same way as worker in producing the
> > > > results if the results from workers is not available.
> > >
> > > We can make backend do the same work as any worker, but that
> > > will complicate the code for less than 2 % perfomance boost.
> >
> > Why do you think it will be just 2%? I think for single worker case,
> > it should be much more as the master backend will be less busy in
> > consuming tuples from tuple queue. I can't say much about
> > code-complexity, as I haven't yet looked carefully at the logic of
> > patch, but we didn't find much difficulty while doing it for parallel
> > scans. One of the commit which might help you in understanding how
> > currently heap scans are parallelised is
> > ee7ca559fcf404f9a3bd99da85c8f4ea9fbc2e92, you can see if that can
> > help you in anyway for writing a generic API for Gin parallel builds.
>
> I looked at the timing details some time ago, which showed
> that the backend spent about 1% of total time on data
> transfer from 1 worker, and 3% on transfer and merging from
> 2 workers. So if we use (active backend + 1 worker) instead
> of (passive backend + 2 workers), we still have to spend
> 1.5% on transfer and merging.
>

I think here the comparison should be between the case of (active backend +
1 worker) with (passive backend + 1 worker) or (active backend + 2 worker)
with (passive backend + 2 workers). I don't think it is good assumption
that workers are always freely available and you can use them as and when
required for any operation.

>
> Or we can look at these measurements (from yesterday's
> message):
>
> wnum mem(MB) time(s)
> 0 16 247
> 1 16 256
> 2 16 126
>
> If 2 workers didn't have to transfer and merge their
> results, they would have finished in 247 / 2 = 123.5
> seconds. But the transfer and merging took another 2.5
> seconds. The merging takes a little longer than the
> transfer. If we now use backend+worker we get rid of 1
> transfer, but still have to do 1 transfer and then merge, so
> we will save less than a quarter of those 2.5 seconds.
>

If I understand the above data correctly, then it seems to indicate that
majority of the work is done in processing the data, so I think it should
be better if master and worker both can work together.

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com