Re: Restructured Shared Buffer Hash Table

On Tue, 7 Jul 2026 at 20:14, Heikki Linnakangas <hlinnaka(at)iki(dot)fi> wrote:
>
> On 07/07/2026 21:41, Dhruv Aron wrote:
> > At Databricks, we’ve found that the existing dynahash table structure is
> > leaving performance gains on the table when it comes to shared buffer
> > lookups: the multi-level structure (directory, segment, bucket chain,
> > freelist) appears excessive for the shared buffers and could be
> > simplified to boost performance and lower memory overhead. As such, we
> > are proposing a specialized hash table just for this purpose and would
> > appreciate feedback on this approach.
>
> Yeah, the dynahash has features that we just don't need in the buffer
> lookup table...
>
> The indirection with the directory is actually unnecessary for all the
> shared memory hash tables, since none of them can be resized. I've
> wondered if we should try to eliminate that from dynahash for all shmem
> hash tables. But the buffer lookup table is very performance-critical,
> so if there are any performance gains to be had, it's indeed probably
> worthwhile to have a separate implementation just for it.
>
> > bufmgr.c also changed slightly to prevent a race condition.
>
> Hmm, we're now holding the buffer header lock much longer than before,
> in InvalidateBuffer(). It's a spinlock, it really should not be held for
> more than a few instructions. BufTableDelete() is very fast in the new
> implementation, but still. Could we perhaps do some of
> BufTableDelete()'s work ahead of time, before we acquire the buffer
> header lock? Or maybe it's not a problem, in which case some kind of a
> worst case scenario benchmark to show that would be nice. Maybe test how
> it behaves when you have a lot of hash collisions, I think that'd make
> BufTableDelete() more expensive.
>
> > My testing
> > (helper script also attached) indicates that all three standard hash
> > table operations (insert, lookup, and delete) generally execute
> > significantly faster than the existing PG18 dynahash counterparts:
>
> Nice!
>
> I wonder how big the impact is with real world workloads. I've certainly
> seen the buffer table lookups consume a fair share of CPU time, so I'd
> assume that it shows up.
>
>
> Another data point:
>
> unpatched master, with shared_buffers='128 MB':
>
> postgres=# select * from pg_shmem_allocations where name like 'Shared
> Buffer%' order by name ;
> name | off | size | allocated_size
> ----------------------------+-----------+--------+----------------
> Shared Buffer Lookup Table | 141607040 | 926000 | 926108
> (1 row)
>
> With this patch:
>
> postgres=# select * from pg_shmem_allocations where name like 'Shared
> Buffer%' order by name ;
> name | off | size | allocated_size
> ------------------------------+-----------+--------+----------------
> Shared Buffer Lookup Buckets | 141607040 | 65536 | 65644
> Shared Buffer Lookup Entries | 141672576 | 393216 | 393216
> (2 rows)
>
> So the new hash table takes much less memory. That's nice because you
> can then fit more in CPU caches.

Where did this thread come from? I can't see any conversation beyond
this single email, and as such, can't see any patch either.

Thom