Re: Better shared data structure management and resizable shared data structures

From: Heikki Linnakangas <hlinnaka(at)iki(dot)fi>
To: Ashutosh Bapat <ashutosh(dot)bapat(dot)oss(at)gmail(dot)com>, Andres Freund <andres(at)anarazel(dot)de>
Cc: pgsql-hackers <pgsql-hackers(at)postgresql(dot)org>, chaturvedipalak1911(at)gmail(dot)com
Subject: Re: Better shared data structure management and resizable shared data structures
Date: 2026-03-06 14:12:57
Message-ID: 26c766d6-db0f-43d3-a618-44f8d40a3121@iki.fi
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

I spent some time cleaning up the new registration machinery. I didn't
look at the "resizeable" part yet, but it should fit in nicely, as
Ashutosh demonstrated.

I added a lot of comments, changed the existing docs on how to allocate
shared memory in extensions to use the machinery, and tons of other
cleanups. I'm getting pretty happy with this, but there are a couple of
weak spots:

Firstly, I'm not sure what to do with ShmemRegisterHash() and the
'HASHCTL *infoP' argument to it. I feel it'd be nicer if the HASHCTL was
just part of the ShmemHashDesc struct, but I'm not sure if that fits all
the callers. I'll have to try that out I guess.

Secondly, I'm not 100% happy with the facilities we provide to
extensions. The lifecycle of _PG_init() and shmem_request/startup_hooks
is a little messy. The status quo is that a shared library gets control
in three different places:

1. _PG_init() gets called early at postmaster startup, if the library is
in shared_preload_libraries. If it's not in shared_preload_libraries, it
gets called whenever the module is loaded.

2. The library can install a shmem_request_hook, which gets called early
at postmaster startup, but after initializing the MaxBackends GUC. It
only gets called when the library is loaded via shared_preload_libraries.

3. The library can install a shmem_startup_hook. It gets called later at
postmaster startup, after the shared memory segment has been allocated.
In EXEC_BACKEND mode it also gets called at backend startup. It does not
get called if the library is not listed in shared_preload_libraries.

None of these is quite the right moment to call the new
ShmemRegisterStruct() function. _PG_init() is too early if the extension
needs MaxBackends for sizing the shared memory area. shmem_request_hook
is otherwise good, but in EXEC_BACKEND mode, the ShmemRegisterStruct()
function needs to also be called backend startup and shmem_request_hook
is not called at backend startup. shmem_startup_hook() is too late.

For now, I documented that an extension should call
ShmemRegisterStruct() from _PG_init(), but may adjust the size in the
shmem_request_hook, if needed.

Another wrinkle here is that you still need the shmem_request_hook, if
you want to call RequestNamedLWLockTranche(). It cannot be called from
_PG_init(). I'm not sure why that is.

So I think that requires a little more refactoring, I think an extension
should need to use shmem_request/startup_hook with the new APIs anymore.
It should provide more ergonomic callbacks or other mechanisms to
accomplish the same things.

- Heikki

Attachment Content-Type Size
v2-0001-Fix-pointer-type-of-ShmemAllocatorData-index.patch text/x-patch 839 bytes
v2-0002-Introduce-a-new-mechanism-for-registering-shared-.patch text/x-patch 50.9 KB
v2-0003-Convert-pg_stat_statements-to-use-the-new-interfa.patch text/x-patch 6.5 KB
v2-0004-Use-the-new-mechanism-in-a-few-core-subsystems.patch text/x-patch 34.4 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Boris Mironov 2026-03-06 14:21:36 Re: Idea to enhance pgbench by more modes to generate data (multi-TXNs, UNNEST, COPY BINARY)
Previous Message Chao Li 2026-03-06 14:10:42 Re: Mis-use of type BlockNumber?