Re: Rewrite sinval messaging to reduce contention

I wrote:
> ... enabling LWLOCK_STATS shows that the contention
> rate on the sinval locks is now completely negligible --- one block
> per thousand acquisitions on SInvalWriteLock, and less than one in
> 10000 on SInvalReadLock. The vast majority of the LWLock contention
> now comes from WALInsertLock and the LockMgr locks:

> Lock # acquisitions # times blocked

> SInvalReadLock 6469840 380
> SInvalWriteLock 240567 163
> WALInsertLock 2388805 89142
> LockMgr partition locks 8253142 177715

For comparison's sake I rebuilt CVS HEAD with LWLOCK_STATS enabled
and repeated the same test. I got

SInvalLock 81090044 505750
WALInsertLock 2382254 62747
LockMgr locks 10657480 171799

The change in sinval numbers is gratifying, but for awhile I didn't
believe these results because of the discrepancy in LockMgr acquisition
figures. I think though that what we are seeing here is that CVS HEAD
has the reset-everyone-on-sinval-queue-overflow behavior, which results
in a whole lot of useless cache resets, which results in a lot of
unnecessary cache reloads, and every one of those requires taking
AccessShareLock on one or more system catalogs in order to suck the data
back in. So the reduction in LockMgr traffic is explained by not doing
so many cache resets.

regards, tom lane