Re: Buffers from parallel workers not accumulated to upper nodes with gather merge

On Tue, 21 Jul 2020 17:34:57 +0200
Jehan-Guillaume de Rorthais <jgdr(at)dalibo(dot)com> wrote:
> On Mon, 20 Jul 2020 15:29:00 +0530
> Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> > On Mon, Jul 20, 2020 at 1:29 PM Jehan-Guillaume de Rorthais
> > <jgdr(at)dalibo(dot)com> wrote:
> > > On Mon, 20 Jul 2020 11:30:34 +0530
> > > Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
[...]
> > > > I think irrespective of whether we want to call
> > > > ExecShutdownGatherMergeWorkers in gather_merge_getnext (when we know
> > > > we are done aka binaryheap_empty) to fix this particular issue, it is
> > > > better to shutdown the workers as soon as we are done similar to what
> > > > we do for Gather node. It is good to release resources as soon as we
> > > > can.
> > >
> > > Indeed. I was focusing on the bug and I didn't thought about that. The
> > > patch I test was really just a quick test. I'll have a closer look at
> > > this, but I suppose this might be considered as a separate commit?
> > >
> >
> > Good Question. Initially, I thought we will have it in a same commit,
> > but now on again thinking about it, we might want to keep this one for
> > the HEAD only and ExecShutdownNode related change in the back-branches.
>
> OK.
>
> I'll send a patch proposal for Gather Merge on HEAD tomorrow.

Please, find in attachment the patch to release Gather Merge workers as soon as
they are exhausted.

I was wondering how interesting it would be to release each worker individually
of each others? This might be interesting for long running queries and/or large
number of workers. This would apply to Gather as well. Do you think it worth to
give it try? Maybe it has already been discussed in the past?

Regards,