Re: Rewrite sinval messaging to reduce contention

From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: pgsql-patches(at)postgresql(dot)org
Subject: Re: Rewrite sinval messaging to reduce contention
Date: 2008-06-19 02:37:19
Message-ID: 12066.1213843039@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-patches

I wrote:
> In this thread back in January
> http://archives.postgresql.org/pgsql-performance/2008-01/msg00001.php
> we saw evidence that contention for SInvalLock was a problem,
> and we (or at least I) concluded that the sinval messaging system
> needed a nontrivial rewrite to get rid of its habit of sending everyone
> to read the queue at the same time. Here is the proposed rewrite.

After further thought, I came up with two more improvements:

* It turns out not to be too hard to apply the idea of moving multiple
messages per LWLock acquisition on the writing side too, so I did that.

* There is not actually any good reason why writers have to block
readers or vice versa, except in the infrequent case that SICleanupQueue
is needed. Writers don't look at the ProcStates that the readers are
changing. The only point of overlap is that a writer will change
maxMsgNum which readers want to look at --- but we are already assuming
in several places that reading or writing an int is atomic, and if we
assume that here too, it seems to work fine. To implement this, I split
SInvalLock into SInvalReadLock and SInvalWriteLock.

This seems to be reaching a point of diminishing returns for the
particular test case I'm using --- the TPS rate only went up from 700
to 730 or so. However, enabling LWLOCK_STATS shows that the contention
rate on the sinval locks is now completely negligible --- one block
per thousand acquisitions on SInvalWriteLock, and less than one in
10000 on SInvalReadLock. The vast majority of the LWLock contention
now comes from WALInsertLock and the LockMgr locks:

Lock # acquisitions # times blocked

SInvalReadLock 6469840 380
SInvalWriteLock 240567 163
WALInsertLock 2388805 89142
LockMgr partition locks 8253142 177715

So I think this patch accomplishes the goal of making sinval not be a
cause of contention.

Patch version 2 attached.

regards, tom lane

Attachment Content-Type Size
sinval-rewrite-2.patch.gz application/octet-stream 11.4 KB

In response to

Responses

Browse pgsql-patches by date

  From Date Subject
Next Message Tom Lane 2008-06-19 05:28:43 Re: Rewrite sinval messaging to reduce contention
Previous Message Simon Riggs 2008-06-18 22:41:05 Re: [HACKERS] Hint Bits and Write I/O