Re: group locking: incomplete patch, just for discussion

On Thu, Nov 13, 2014 at 4:57 PM, Jeff Davis <pgsql(at)j-davis(dot)com> wrote:
> On Thu, Nov 13, 2014 at 11:26 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>> On Thu, Nov 13, 2014 at 3:38 AM, Jeff Davis <pgsql(at)j-davis(dot)com> wrote:
>> > If two backends both have an exclusive lock on the relation for a join
>> > operation, that implies that they need to do their own synchronization,
>> > because obviously the lock manager is not doing it for them.
>>
>> This doesn't make sense to me. Why would they need to synchronize
>> access to a relation in order to join it?
>
> Unfortunate typo: that was supposed to be "joint" operation, just meaning
> that they are working together for something (e.g. CLUSTER, VACUUM FULL as
> you suggest). Sorry for the confusion.

Ah, well, that's different.

> I meant that the backends need to divide up the work somehow. And if each
> operator needs to divide up the work before operating, that means we need to
> change every operator.

I think I see your point, which it just so happens Amit articulated to
me in different words while we were chatting about this problem this
morning. We want to avoid waiving the mutual exclusion provided by
the lock manager only to end up re-implementing something very much
like the current lock manager to arbitrate resource contention among
backends within a parallel group. However, we also don't want the
mutual exclusion that the lock manager provides to serve as a barrier
to implementing useful parallelism; that is, if the parallel backends
want to manage conflicts themselves instead of letting the lock
manager do it, that should be possible.

In http://www.postgresql.org/message-id/CA+TgmoYGpLoJo+LG1beBOs8gdjwjTQ0qdmxsYJ4ihFyJ11Tr-g@mail.gmail.com
I propose a new approach. The basic idea is that any heavyweight lock
that we hold at the start of a parallel phase can't represent a bona
fide conflict between the activities of two different backends, so we
arrange to waive conflicts over those locks. The problem (as Andres
points out) is that those locks could be subsequently re-taken by
operations that are not parallel-safe, and the new locks would be
granted locally regardless of the shared lock table because the
current backend would already hold them. However, I think it's fine
to make it the job of the parallelism infrastructure to plug that
hole, rather than trying to enforce it in the lock manager. There are
LOTS of things that need to be prohibited in parallel mode and only
allowed once they've been made sufficiently safe, and in the first
version, that's certainly going to include - among other things - any
operation that writes data. I think it's OK to the job of the people
who want to relax those prohibitions to deal with making it safe to do
so.

To reiterate the basic problem here, if we do nothing at all about the
lock manager, a parallel backend can stall trying to grab an
AccessExclusiveLock that the user backend alread holds, either because
the user backend holds an AccessExclusiveLock as well, or because some
other process is waiting for one, we'll deadlock and not notice. We
have to do *something* about that. I'd like to arrive at some
consensus on how to move forward here as soon as possible, because the
clock is ticking here.
http://www.postgresql.org/message-id/CA+TgmoYGpLoJo+LG1beBOs8gdjwjTQ0qdmxsYJ4ihFyJ11Tr-g@mail.gmail.com
articulates what I currently believe to be the best plan.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company