Re: Reducing overhead of frequent table locks

On Wed, May 25, 2011 at 3:35 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Simon Riggs <simon(at)2ndQuadrant(dot)com> writes:
>> On Wed, May 25, 2011 at 1:44 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>>> On Wed, May 25, 2011 at 8:27 AM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
>>>> Design seemed relatively easy from there: put local lock table in
>>>> shared memory for all procs. We then have a use_strong_lock at proc
>>>> and at transaction level. Anybody that wants a strong lock first sets
>>>> use_strong_lock at proc and transaction level, then copies all local
>>>> lock data into shared lock table,
>
>>> I'm not following this...
>
>> Which bit aren't you following? It's a design outline for how to
>> implement, deliberately brief to allow a discussion of design
>> alternatives.
>
> What I'm not following is how moving the local lock table into shared
> memory can possibly be a good idea.  The reason we invented the local
> lock table in the first place (we didn't use to have one) is so that a
> process could do some manipulations without touching shared memory.
> (Notably, it is currently nearly free, and certainly lock-free, to
> re-request a lock type you already hold.  This is not an infrequent
> case.)  That advantage will go away if you do this.

The basis for this is that weak locks do not conflict with each other,
whereas strong locks conflict with both strong and weak locks.
(There's a couple of special cases which I ignore for now).
(Using Robert's description of strong/weak locks)

Since most actions in normal running only require weak locks then we
see that 99% of the time we don't need to share lock information at
all.

So the idea is that we have 2 modes of operation: mode (1) when nobody
is requesting a strong lock we don't share lock information. We switch
into mode (2) when somebody requests a strong lock and in this mode we
must share all lock information just as we do now.

The basic analysis is that we have a way of removing 99% of the
overhead of lock information sharing. We still have the local lock
table and we still perform locking, we just don't share that with
other backends unless we need to. So there is a slight reduction in
path length and a total avoidance of contention.

Ideally, we would want to be in mode 2 for a short period of time.

The difficulty is how to move from mode 1 (non-shared locking) to mode
2 (shared locking) and back again?

A strong lock request causes the mode flip automatically via one of
these mechanisms:
1. signal to each backend causes them to update shared lock
information (at that point non-conflicting)
2. local lock table in shared memory
3. files
4. other
The requirement is that the mode be flipped in all backends before we
process the request for a strong lock.

The idea is to make the local lock table accessible for occasional use
in mode switching. Reading the local lock table by its owning backend
would always be lock free. Locks are only required when modifying the
local lock table by the owning backend, or when another backend reads
it. So making the local lock table accessible is not a problem.

Flipping back from mode 2 to mode 1 should be fairly slow, since DDL
usually occurs in groups.

--
 Simon Riggs                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services