Re: Changing shared_buffers without restart

On 09/02/2026 17:15, Ashutosh Bapat wrote:
> On Sun, Feb 8, 2026 at 5:14 AM Heikki Linnakangas <hlinnaka(at)iki(dot)fi> wrote:
>> Could you write a standalone test module in src/test/modules to
>> demonstrate how to use the resizable shmem segments, please? That'd
>> allow focusing on that interface without worrying all the other
>> complexities of shared buffers.
>
> From your writeup it seems like you are leaning towards creating the
> shared memory segments on-demand rather than having predefined
> segments as done in the patch.

Right. Same as with normal shared memory structs: there are
ShmemInitStruct calls scattered throughout the codebase, there's no need
to predefine them in a single header file or anything like that.

(Note: "on-demand" doesn't mean "on the fly", i.e. each shared memory
area still needs to be registered at postmaster startup.)

> I think your proposal is interesting
> and might create a possibility for extensions to be able to create
> resizable shared memory structures.

Ah yes, good point. This facility should certainly be open for
extensions too. I didn't even realize that the predefined segments
scheme would not allow that.

> Since the shared memory segments are predefined, a test module, as you
> suggest, does not have a segment that it can use. That led me to think
> that you are imagining some kind of on-demand shared memory segments
> that extensions or test modules can use. I think that will be a useful
> feature by itself. Just as an example, imagine an extension to provide
> shared plan cache which resizes the plan cache as needed. However, we
> need to make sure that these on-demand shared memory segments work
> well with CalculateShmemSize(), PGSharedMemoryDetach(),
> AnonymousShmemDetach() etc.

+1

> 2. There is no predefined limit on the number of segments. When
> CalculateShmemSize() is called, BufferManagerShmemSize() outputs the
> shared memory required in the main segment and total of shared memory
> required in the other segments. Similarly shmem_request_hook of each
> external module outputs the shared memory required in the main segment
> and the total of shared memory required in the other segments. The
> main segment is created in CreateSharedMemoryAndSemaphores() directly
> using the requested size. The other output size is merely used for
> reporting purposes in InitializeShmemGUCs(). When allocating shared
> data structures, the main shared memory data structures are allocated
> using ShmemInitStruct(), whereas the data structures in other memory
> segments are allocated using ShmemInitStructExt() which takes
> immediate allocation size and size of address space to be reserved as
> arguments. It does not require segment_id though. For every new data
> structure that ShmemInitStructExt() encounters, it a. creates a new
> shared memory segment using PGSharedMemoryCreate(), b. allocates that
> structure with the initial size. This function also needs to create
> the PGShmemInfo and AnonShmemData entries corresponding to new
> segments and make them available to PGSharedMemoryDetach(),
> AnonymousShmemDetach() etc. For that we create a shared hash table in
> the main shared memory segment (just like ShmemIndex) where we store
> the metadata against each segment name (by coining it from the name of
> the structure). We expect ShmemInitStructExt() to allocate structures
> and segments only in the Postmaster and only at the beginning. When a
> backend starts, it pulls the segment metadata from the shared hash
> table which is ultimately used by PGSharedMemoryDetach(),
> AnonymousShmemDetach(). At run time any backend which has access to
> the shared memory should be able to call ShmemResizeStruct() given a
> resizable structure and new size. Some higher level synchronization is
> needed to ensure that the same structure is not resized simultaneously
> by two backends.
>
> The first approach is simple but has limited use given the fixed
> number of segments. Second is more flexible but that's some work. I am
> not sure whether it's worth doing all that if there are hardly any
> extensions which could use resizable shared data structures.

It doesn't seem *that* much more work.

Putting this patch aside for a moment, I don't much like our current
interface for defining shared memory structs anyway. The
[SubSystem]ShmemSize() functions feel too detached from the
ShmemInitStruct() calls. For example, it's an easy mistake to return a
slightly different size in the FoobarShmemSize() call than what you use
in the ShmemInitStruct() call. And then there's the fact that the
initialization functions run both at postmaster startup but also at
backend startup in EXEC_BACKEND mode. There were reasons for that when
EXEC_BACKEND was introduced, but it's always felt awkward to me.

I feel that it'd be good to have a single definition of each shmem
struct, and derive all the other things from there.

Attached is a proof-of-concept of what I have in mind. Don't look too
closely at how it's implemented, it's very hacky and EXEC_BACKEND mode
is slightly broken, for example. The point is to demonstrate what the
callers would look like. I converted only a few subsystems to use the
new API, the rest still use ShmemInitStruct() and ShmemInitHash().

With this, initialization of a subsystem that defines a shared memory
area looks like this:

--------------

/* This struct lives in shared memory */
typedef struct
{
int field;
} FoobarSharedCtlData;

static void FoobarShmemInit(void *arg);

/* Descriptor for the shared memory area */
ShmemStructDesc FoobarShmemDesc = {
.name = "Foobar subsystem",
.size = sizeof(FoobarSharedCtlData),
.init_fn = FoobarShmemInit,
};

/* Pointer to the shared memory struct */
#define FoobarCtl ((FoobarSharedCtlData *) FoobarShmemDesc.ptr)

/*
* Register the shared memory struct. This is called once at
* postmaster startup, before the shared memory segment is allocated,
* and in EXEC_BACKEND mode also early at backend startup.
*
* For core subsystems, there's a list of all these functions in core
* in ipci.c, similar to all the *ShmemSize() and *ShmemInit() functions
* today. In an extension, this would be done in _PG_init() or in
* the shmem_request_hook, replacing the RequestAddinShmemSpace calls
* we have today.
*/
void
FoobarShmemRegister(void)
{
ShmemRegisterStruct(&FoobarShmemDesc);
}

/*
* This callback is called once at postmaster startup, to initialize
* the shared memory struct. FoobarShmemDesc.ptr has already been
* set when this is called.
*/
static void
FoobarShmemInit(void *arg)
{
memset(FoobarCtl, 0, sizeof(FoobarSharedCtlData));
FoobarCtl->field = 123;
}

--------------

The ShmemStructDesc provides room for extending the facility in the
future. For example, you could specify alignment there, or an additional
"attach" callback when you need to do more per-backend initialization in
EXEC_BACKEND mode. And with the resizeable shared memory, a max size.

Thoughts?

- Heikki