Re: sinval synchronization considered harmful

On Thu, Jul 21, 2011 at 4:54 PM, Noah Misch <noah(at)2ndquadrant(dot)com> wrote:
> On Wed, Jul 20, 2011 at 09:46:33PM -0400, Robert Haas wrote:
>> For the last week or so, in between various other tasks, I've been
>> trying to understand why performance drops off when you run pgbench -n
>> -S -c $CLIENTS -j $CLIENTS -T $A_FEW_MINUTES at very high client
>> counts.  The answer, in a word, is SIGetDataEntries().  I believe we
>> need to bite the bullet and rewrite this using a lock-free algorithm,
>> using memory barriers on processors with weak memory ordering.
>> Perhaps there is another way to do it, but nothing I've tried has
>> really worked so far, and I've tried quite a few things.  Here's the
>> data.
>>
>> On unpatched master, performance scales pretty much linearly out to 32
>> clients.  As you add more clients, it drops off:
>
>> [80 clients]
>> tps = 132518.586371 (including connections establishing)
>> tps = 130968.749747 (including connections establishing)
>> tps = 132574.338942 (including connections establishing)
>
>> [80 clients, with lazy vxid locks and sinval-unlocked]
>> tps = 203256.701227 (including connections establishing)
>> tps = 190637.957571 (including connections establishing)
>> tps = 190228.617178 (including connections establishing)
>
> Nice numbers.  The sinval-unlocked.patch implementation looks like it's taking a
> sound direction.
>
> In
> http://archives.postgresql.org/message-id/CA+TgmobbxMh_9zjudheSWO6m8sBMb5hdZt+3ChCLuv5eztv8Ug@mail.gmail.com,
> you quoted 210k TPS when you stubbed out AcceptInvalidationMessages().  Is it
> correct to conclude that AcceptInvalidationMessages() still reduces the
> transaction rate by 5-10% with this stack of patches?

Good question - I have not tested.

One idea I just had... if we use a 64-bit counter for maxMsgNum, maybe
we could make AcceptInvalidationMessages() a macro, something like
this:

if (MyProcState->nextMsgNum != shmInvalState->maxMsgNum)
ReallyAcceptInvalidationMessages();

That ought to be extremely cheap and - if we use 64-bit counters for
the message-number counters - safe. You might object that the load of
maxMsgNum might migrate backward, but it can't possibly back up any
further than the preceding lock acquisition, since that's required to
be a full memory barrier on every architecture. And if we haven't
acquired a relevant lock, then a relevant sinval message could show up
an instance after we check regardless of the implementation.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company