Re: Changing shared_buffers without restart

On Tue, Feb 10, 2026 at 3:44 AM Heikki Linnakangas <hlinnaka(at)iki(dot)fi> wrote:
>
> On 09/02/2026 17:15, Ashutosh Bapat wrote:
> > On Sun, Feb 8, 2026 at 5:14 AM Heikki Linnakangas <hlinnaka(at)iki(dot)fi> wrote:
> >> Could you write a standalone test module in src/test/modules to
> >> demonstrate how to use the resizable shmem segments, please? That'd
> >> allow focusing on that interface without worrying all the other
> >> complexities of shared buffers.
> >
> > From your writeup it seems like you are leaning towards creating the
> > shared memory segments on-demand rather than having predefined
> > segments as done in the patch.
>
> Right. Same as with normal shared memory structs: there are
> ShmemInitStruct calls scattered throughout the codebase, there's no need
> to predefine them in a single header file or anything like that.
>
> (Note: "on-demand" doesn't mean "on the fly", i.e. each shared memory
> area still needs to be registered at postmaster startup.)

Yes, that's what I have in mind.

>
> > I think your proposal is interesting
> > and might create a possibility for extensions to be able to create
> > resizable shared memory structures.
>
> Ah yes, good point. This facility should certainly be open for
> extensions too. I didn't even realize that the predefined segments
> scheme would not allow that.
>
> > Since the shared memory segments are predefined, a test module, as you
> > suggest, does not have a segment that it can use. That led me to think
> > that you are imagining some kind of on-demand shared memory segments
> > that extensions or test modules can use. I think that will be a useful
> > feature by itself. Just as an example, imagine an extension to provide
> > shared plan cache which resizes the plan cache as needed. However, we
> > need to make sure that these on-demand shared memory segments work
> > well with CalculateShmemSize(), PGSharedMemoryDetach(),
> > AnonymousShmemDetach() etc.
>
> +1
>
> > 2. There is no predefined limit on the number of segments. When
> > CalculateShmemSize() is called, BufferManagerShmemSize() outputs the
> > shared memory required in the main segment and total of shared memory
> > required in the other segments. Similarly shmem_request_hook of each
> > external module outputs the shared memory required in the main segment
> > and the total of shared memory required in the other segments. The
> > main segment is created in CreateSharedMemoryAndSemaphores() directly
> > using the requested size. The other output size is merely used for
> > reporting purposes in InitializeShmemGUCs(). When allocating shared
> > data structures, the main shared memory data structures are allocated
> > using ShmemInitStruct(), whereas the data structures in other memory
> > segments are allocated using ShmemInitStructExt() which takes
> > immediate allocation size and size of address space to be reserved as
> > arguments. It does not require segment_id though. For every new data
> > structure that ShmemInitStructExt() encounters, it a. creates a new
> > shared memory segment using PGSharedMemoryCreate(), b. allocates that
> > structure with the initial size. This function also needs to create
> > the PGShmemInfo and AnonShmemData entries corresponding to new
> > segments and make them available to PGSharedMemoryDetach(),
> > AnonymousShmemDetach() etc. For that we create a shared hash table in
> > the main shared memory segment (just like ShmemIndex) where we store
> > the metadata against each segment name (by coining it from the name of
> > the structure). We expect ShmemInitStructExt() to allocate structures
> > and segments only in the Postmaster and only at the beginning. When a
> > backend starts, it pulls the segment metadata from the shared hash
> > table which is ultimately used by PGSharedMemoryDetach(),
> > AnonymousShmemDetach(). At run time any backend which has access to
> > the shared memory should be able to call ShmemResizeStruct() given a
> > resizable structure and new size. Some higher level synchronization is
> > needed to ensure that the same structure is not resized simultaneously
> > by two backends.
> >
> > The first approach is simple but has limited use given the fixed
> > number of segments. Second is more flexible but that's some work. I am
> > not sure whether it's worth doing all that if there are hardly any
> > extensions which could use resizable shared data structures.
>
> It doesn't seem *that* much more work.
>
> Putting this patch aside for a moment, I don't much like our current
> interface for defining shared memory structs anyway. The
> [SubSystem]ShmemSize() functions feel too detached from the
> ShmemInitStruct() calls. For example, it's an easy mistake to return a
> slightly different size in the FoobarShmemSize() call than what you use
> in the ShmemInitStruct() call. And then there's the fact that the
> initialization functions run both at postmaster startup but also at
> backend startup in EXEC_BACKEND mode. There were reasons for that when
> EXEC_BACKEND was introduced, but it's always felt awkward to me.
>
> I feel that it'd be good to have a single definition of each shmem
> struct, and derive all the other things from there.
>
> Attached is a proof-of-concept of what I have in mind. Don't look too
> closely at how it's implemented, it's very hacky and EXEC_BACKEND mode
> is slightly broken, for example. The point is to demonstrate what the
> callers would look like. I converted only a few subsystems to use the
> new API, the rest still use ShmemInitStruct() and ShmemInitHash().
>
> With this, initialization of a subsystem that defines a shared memory
> area looks like this:

I think this is much better than what we have today. I haven't tried
adding APIs to specify resizable structures, but I will try that
tomorrow unless you want me to wait for this to mature.

I didn't look too deeply but what's broken in the EXEC_BACKEND case?

>
> --------------
>
> /* This struct lives in shared memory */
> typedef struct
> {
> int field;
> } FoobarSharedCtlData;
>
> static void FoobarShmemInit(void *arg);
>
> /* Descriptor for the shared memory area */
> ShmemStructDesc FoobarShmemDesc = {
> .name = "Foobar subsystem",
> .size = sizeof(FoobarSharedCtlData),
> .init_fn = FoobarShmemInit,
> };
>
> /* Pointer to the shared memory struct */
> #define FoobarCtl ((FoobarSharedCtlData *) FoobarShmemDesc.ptr)
>

I don't like this much since it limits the ability to debug. A macro
is not present in the symbol table. How about something like attached?
Once we do that we can make the ShmemStructDesc local to
ShmemRegisterStruct() calls and construct them on the fly. There's no
other use for them.

> /*
> * Register the shared memory struct. This is called once at
> * postmaster startup, before the shared memory segment is allocated,
> * and in EXEC_BACKEND mode also early at backend startup.
> *
> * For core subsystems, there's a list of all these functions in core
> * in ipci.c, similar to all the *ShmemSize() and *ShmemInit() functions
> * today. In an extension, this would be done in _PG_init() or in
> * the shmem_request_hook, replacing the RequestAddinShmemSpace calls
> * we have today.
> */
> void
> FoobarShmemRegister(void)
> {
> ShmemRegisterStruct(&FoobarShmemDesc);
> }
>
> /*
> * This callback is called once at postmaster startup, to initialize
> * the shared memory struct. FoobarShmemDesc.ptr has already been
> * set when this is called.
> */
> static void
> FoobarShmemInit(void *arg)
> {
> memset(FoobarCtl, 0, sizeof(FoobarSharedCtlData));
> FoobarCtl->field = 123;
> }
>
> --------------
>
> The ShmemStructDesc provides room for extending the facility in the
> future. For example, you could specify alignment there, or an additional
> "attach" callback when you need to do more per-backend initialization in
> EXEC_BACKEND mode. And with the resizeable shared memory, a max size.
>
> Thoughts?

LGTM.

One more comment.
/* estimated size of the shmem index table (not a hard limit) */
#define SHMEM_INDEX_SIZE (64)

What do you mean by "not a hard limit"?

We will be limited by the number of entries in the array; a limit that
doesn't exist in the earlier implementation. I mean one can easily
have 100s of extensions loaded, each claiming a couple shared memory
structures. How large do we expect the array to be? Maybe we should
create a linked list which is converted to an array at the end of
registration. The array and its size can be easily passed to the child
through the launch backend as compared to the list. Looking at
launch_backend changes, even that doesn't seem to be required since
SubPostmasterMain() calls RegisterShmemStructs().

Also, do we want to discuss this in a thread of its own?

--
Best Wishes,
Ashutosh Bapat