Re: Restructured Shared Buffer Hash Table

On Tue, 7 Jul 2026 at 20:14, Heikki Linnakangas <hlinnaka(at)iki(dot)fi> wrote:
> Or maybe it's not a problem, in which case some kind of a
> worst case scenario benchmark to show that would be nice. Maybe test how
> it behaves when you have a lot of hash collisions, I think that'd make
> BufTableDelete() more expensive.

My additional benchmarking would suggest that, on average (one entry per
bucket), the spinlock would be held for an extra ~50ns compared to
releasing the header lock earlier for the current dynahash implementation:

Bucket Chain Length

Average Time Under Spinlock for Dynahash Implementation (ns)

Average Time Under Spinlock for New Table, Releasing Lock After Delete (ns)

1

64.4

112.9

2

63.4

118.6

4

63.1

122.3

8

68.8

139.8

16

76.0

159.1

32

77.7

195.0

64

102.0

248.6

On Tue, Jul 7, 2026 at 8:28 PM Ashutosh Bapat <ashutosh(dot)bapat(dot)oss(at)gmail(dot)com>
wrote:
> If we are using a different data structure for buffer lookup,
> it will be good to consider ease of resizing as well.

Yes, we have internal functionality to resize (both shrink and expand) this
restructured table without rehashing the entire table, acquiring a global
lock, or introducing new synchronization primitives. Will create and share
a follow-up patch going into more detail soon.

Thanks,
Dhruv Aron

On Tue, Jul 7, 2026 at 8:28 PM Ashutosh Bapat <ashutosh(dot)bapat(dot)oss(at)gmail(dot)com>
wrote:

> On Wed, Jul 8, 2026 at 12:44 AM Heikki Linnakangas <hlinnaka(at)iki(dot)fi>
> wrote:
> >
> > On 07/07/2026 21:41, Dhruv Aron wrote:
> > > At Databricks, we’ve found that the existing dynahash table structure
> is
> > > leaving performance gains on the table when it comes to shared buffer
> > > lookups: the multi-level structure (directory, segment, bucket chain,
> > > freelist) appears excessive for the shared buffers and could be
> > > simplified to boost performance and lower memory overhead. As such, we
> > > are proposing a specialized hash table just for this purpose and would
> > > appreciate feedback on this approach.
> >
> > Yeah, the dynahash has features that we just don't need in the buffer
> > lookup table...
> >
> > The indirection with the directory is actually unnecessary for all the
> > shared memory hash tables, since none of them can be resized. I've
> > wondered if we should try to eliminate that from dynahash for all shmem
> > hash tables. But the buffer lookup table is very performance-critical,
> > so if there are any performance gains to be had, it's indeed probably
> > worthwhile to have a separate implementation just for it.
>
> For very large buffer pools, the buffer lookup table spans multiple
> memory pages. With buffer resizing capability it will be good to be
> able to resize the buffer lookup table as well. Right now the patch
> does not resize buffer look up table because of its memory layout and
> complexity involved in that operation. If we are using a different
> data structure for buffer lookup, it will be good to consider ease of
> resizing as well.
>
> --
> Best Wishes,
> Ashutosh Bapat
>