Re: Anyone understand shared-memory space usage?

I wrote:
> Does anyone understand the data structures that are allocated in
> shared memory well enough to fix LockShmemSize() properly?

No one volunteered, so I dug into the code and think I have it fixed
now. Leastwise you can run the regression tests even at -N 1 (but
you have to put a "sleep" into regress.sh --- it seems that when you
quit psql, it takes a second or two before the postmaster will accept
another connection. Should backend shutdown take that long??)

It turned out that there were really, really serious problems both in
shared-memory space estimation and in dynahash.c itself. I'm simply
amazed we have not seen more bug reports traceable to running out
of shared memory and/or hashtable errors. Some lowlights:

* One out of every thirty records allocated in a hashtable was simply
being wasted, because the allocator failed to include it in the table's
freelist.

* The routine for expanding a hashtable's top-level directory could
never have worked; I conclude that it's never been executed. (At
default settings it would not be called until the table has exceeded
64K entries, so I can believe we've never seen it run...)

* I think the routine for deleting a hashtable is also broken, because
it individually frees records that it did not allocate individually.
I don't understand why this isn't making the memory management stuff
coredump. Maybe we never free a hashtable?

* Setup of fixed-directory hashtables (ShmemInitHash) was sadly broken;
it's really incredible that it worked at all, because it was (a)
misestimating the size of the space it needed to allocate and then
(b) miscalculating where the directory should be within that space.
As near as I can tell, we have been running with hashtable directories
sitting in space not actually allocated to them. Compared to this,
the fact that the routine also forgot to tell dynahash.c what size
directory it had made hardly matters.

* Several places were estimating the sizes of hashtables using code
that was not quite right (and assumed far more than it should've
about the inner structure of hashtables anyway). Also, having
(mis)calculated the sizes of the major tables in shared memory,
we were requesting a total shared memory block exactly equal to
their sum, with no allowance for smaller data structures (like the
shmem index table) nor any safety factor for estimation error.

I would like someone to check my work; if the code was really as
broken as I think it was, we should have been seeing more problems
than we were. See my changes committed last night in
src/include/utils/hsearch.h
src/backend/utils/hash/dynahash.c
src/backend/storage/ipc/shmem.c
src/backend/storage/ipc/ipci.c
src/backend/storage/buffer/buf_init.c
src/backend/storage/lmgr/lock.c
src/backend/storage/smgr/mm.c

regards, tom lane

PS: I am now wondering whether Daryl Dunbar's problems might not be
due to the shared-memory hash table for locks getting larger than other
people have seen it get. Because of the errors in ShmemInitHash, I
would not be at all surprised to see the system fall over once that
table exceeds 256 entries (or some small multiple thereof).