Re: [HACKERS] Parallel tuplesort (for parallel B-Tree index creation)

On Wed, Jan 17, 2018 at 10:27 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> On Wed, Jan 17, 2018 at 12:27 PM, Peter Geoghegan <pg(at)bowt(dot)ie> wrote:
>> I think that both problems (the live _bt_parallel_scan_and_sort() bug,
>> as well as the general issue with needing to account for parallel
>> worker fork() failure) are likely solvable by not using
>> tuplesort_leader_wait(), and instead calling
>> WaitForParallelWorkersToFinish(). Which you suggested already.
>
> I'm wondering if this shouldn't instead be handled by using the new
> Barrier facilities.

> While I find the Barrier API slightly confusing -- and I suspect I'm
> not entirely alone -- I don't think that's a good excuse for
> reinventing the wheel. The problem of needing to wait for every
> process that does A (in this case, read tuples from the scan) to also
> do B (in this case, finish sorting those tuples) is a very general one
> that is deserving of a general solution. Unless somebody comes up
> with a better plan, Barrier seems to be the way to do that in
> PostgreSQL.
>
> I don't think using WaitForParallelWorkersToFinish() is a good idea.
> That would require workers to hold onto their tuplesorts until after
> losing the ability to send messages to the leader, which doesn't sound
> like a very good plan. We don't want workers to detach from their
> error queues until the bitter end, lest errors go unreported.

What you say here sounds convincing to me. I actually brought up the
idea of using the barrier abstraction a little over a month ago. I was
discouraged by a complicated sounding issue raised by Thomas [1]. At
the time, I figured that the barrier abstraction was a nice to have,
but not really essential. That idea doesn't hold up under scrutiny. I
need to be able to use barriers.

There seems to be some yak shaving involved in getting the barrier
abstraction to do exactly what is required, as Thomas went into at the
time. How should that prerequisite work be structured? For example,
should a patch be spun off for that part?

I may not be the most qualified person for this job, since Thomas
considered two alternative approaches (to making the static barrier
abstraction forget about never-launched participants) without ever
settling on one of them.

[1] https://postgr.es/m/CAEepm=03YnefpCeB=Z67HtQAOEMuhKGyPCY_S1TeH=9a2Rr0LQ@mail.gmail.com
--
Peter Geoghegan