pgsql: Reduce use of heavyweight locking inside hash AM.

Reduce use of heavyweight locking inside hash AM.

Avoid using LockPage(rel, 0, lockmode) to protect against changes to
the bucket mapping. Instead, an exclusive buffer content lock is now
viewed as sufficient permission to modify the metapage, and a shared
buffer content lock is used when such modifications need to be
prevented. This more relaxed locking regimen makes it possible that,
when we're busy getting a heavyweight bucket on the bucket we intend
to search or insert into, a bucket split might occur underneath us.
To compenate for that possibility, we use a loop-and-retry system:
release the metapage content lock, acquire the heavyweight lock on the
target bucket, and then reacquire the metapage content lock and check
that the bucket mapping has not changed. Normally it hasn't, and
we're done. But if by chance it has, we simply unlock the metapage,
release the heavyweight lock we acquired previously, lock the new
bucket, and loop around again. Even in the worst case we cannot loop
very many times here, since we don't split the same bucket again until
we've split all the other buckets, and 2^N gets big pretty fast.

This results in greatly improved concurrency, because we're
effectively replacing two lwlock acquire-and-release cycles in
exclusive mode (on one of the lock manager locks) with a single
acquire-and-release cycle in shared mode (on the metapage buffer
content lock). Testing shows that it's still not quite as good as
btree; for that, we'd probably have to find some way of getting rid
of the heavyweight bucket locks as well, which does not appear
straightforward.

Patch by me, review by Jeff Janes.

Branch
------
master

Details
-------
http://git.postgresql.org/pg/commitdiff/76837c1507cb5a5a0048046233568447729e66dd

Modified Files
--------------
src/backend/access/hash/README | 133 +++++++++++++++++-----------------
src/backend/access/hash/hashinsert.c | 62 ++++++++++------
src/backend/access/hash/hashpage.c | 28 ++------
src/backend/access/hash/hashsearch.c | 61 ++++++++++------
4 files changed, 150 insertions(+), 134 deletions(-)