Re: Shared hash table allocations

On 31/03/2026 01:02, Heikki Linnakangas wrote:
> I wonder if we should change the defaults somehow. In usual
> configurations, people are currently getting much more lock space than
> you'd expect based on max_connections and max_locks_per_transaction, and
> after these patches, they'll get much fewer locks. It might be prudent
> bump up the default max_locks_per_transaction setting so that you'd get
> roughly the same amount of locks in the default configuration.

I did some testing of the memory usage and how removing the wiggle room
affects the number of locks you can acquire. Attached are the test
procedures I used, and proposed patches. The patches are new, designed
to just change the parameters of the hash tables and shmem calculations
with no other changes. They don't include the refactorings we've
discussed so far in this thread. My plan is to commit these new patches
first, and those other refactorings after that. Once these new patches
are committed, the refactorings won't materially change the overall
memory usage or how it's divided between different hash tables, all
those effects are in these new patches.

master: With the default configuration on master, the attached test
procedure can take 14927 locks before hitting "out of shared memory"
error. At that point, all the "wiggle room" is assigned for the LOCK
hash table. A different scenario could make the PROCLOCK hash table
consume all the wiggle room instead, but I believe running out of LOCK
space is more common, and I don't think it changes the big picture
anyway if you hit the ceiling with PROCLOCK instead.

0001: While looking at this, I noticed that we add 10% "safety margin"
to the shmem calculations in predicate.c, but we had already marked the
predicate.c hash tables as HASH_FIXED_SIZE so they were never able to
make use of the safety margin. Oops. The extra memory was available for
the lock.c hash tables, though. After removing that bogus 10% safety
margin from predicate.c, memory usage was reduced by 200 kB, but the
number of locks you could take went down from 14927 to 14159.

0002: As the next step, I also removed the 10% safety margin from
lock.c. That reduced memory usage by another 320 kB, and the number of
locks went down from 14159 to 12815.

0003: After those changes, there's only little extra memory sloshing
around that's not accounted for any data structure. ipci.c reserves a
constant 100 kB, but that's pretty much it. However, there's still
flexibility between the LOCK and the PROCLOCK hash tables. The PROCLOCK
hash table is estimated to be 2x the size of the LOCK table, but when
it's not, the space can get assigned to the LOCK table instead. In patch
0003 I removed that flexibility by marking them both with
HASH_FIXED_SIZE, and making init_size equal to max_size. That also stops
the hash tables from using any of the other remaining wiggle room,
making them truly fixed-size. This doesn't change the overall shared
memory allocated, but the number of locks that the test procedure could
acquire went down from 12815 to 8767, mostly because it cannot "steal"
space from PROCLOCK anymore.

0004: To buy back that lock manager space in common out-of-the box
situations, I propose to bump up the default for
max_locks_per_transactions from 64 to 128. That increases memory usage
again by 3216 kB, making it 2696 kB higher than on master (remember that
the previous changes reduced memory usage). The number of locks you can
take after that is 17535, which more than on master (14927).

Increasing the default won't affect users who have already set
max_locks_per_transaction to a non-default value. They will see that the
number of locks they can acquire with their existing configuration will
be reduced, again because of the lost wiggle room and flexibility
between LOCK and PROCLOCK. Not sure if we could or should do something
about that. Probably best to just document in the release notes that if
you had raised increase max_locks_per_transaction, you might need to
raise it further to be able to accommodate the same amount of locks as
before.

Here's all that in table form:

| Patch | Shmem (kB) | Locks |
| --------------------------------------+------------|-------|
| master | 153560 | 14927 |
| 0001: remove 10% from predicate.c | 153360 | 14159 |
| 0002: remove 10% from lock.c | 153040 | 12815 |
| 0003: Make lock.c tables fixed size | 153040 | 8767 |
| 0004: max_locks_per_transactions=128 | 156256 | 17535 |

This increase in memory usage is not great, but it's not that big in the
grand scheme of things. I think it's well worth, and better than the
sloppy scheme we have today.

Any thoughts, objections?

- Heikki