Re: Better shared data structure management and resizable shared data structures

Hi Haoyu,

On Fri, Jun 12, 2026 at 9:21 PM Haoyu Huang <haoyu(dot)huang(dot)68(at)gmail(dot)com> wrote:
>
> Hi all,
>
> Wanted to introduce myself on this thread and share some related work — with no intention of forking or redirecting what Ashutosh is driving here.
>
> It was great catching up with Ashutosh, David Wein, and Heikki at PGConf Vancouver. We had a working session on resizable shared buffers. It was productive and a lot of fun. The outcome of the session is to surface our work at Databricks on the same topic here.
>
> At Databricks, we have a patch merged in our internal Postgres that enables resizing shared_buffers without restart. It was inspired by Ashutosh's earlier patch on this topic. Heikki and David reviewed it on our side. I'd like to contribute the ideas (and, where useful, the code) back upstream. I want to be explicit that the current series is the path forward. I'd much rather plug into that than propose a competing patchset. Happy to help with review, testing, or specific pieces wherever it's most useful.
>
> Please read the patch as input to the existing effort, not a counter-proposal. Thanks for all the work on this so far.
>
> Here are the major changes we made on top of Ashutosh's earlier patch
>
> Keep one mmap anonymous segment + madvise(MADV_POPULATE_WRITE / MADV_REMOVE) to allocate/free physical pages.
> SB variable names changed to use lowNBuffers, highNBuffers, and maxNBuffers. See the README for more details. We think that this simplifies the code significantly.
> Only the shrink needs to use proc signal barrier to coordinate with all other backends. The other cases are covered by a new AccessNBuffersLock. The coordinator acquires exclusive lock on AccessNBuffersLock when it publishes new buffers. Other backends acquire the shared lock on AccessNBuffersLock when they loop through the buffer array based on the NBuffers value.
> The API to resize the shared buffer is `SELECT pg_resize_shared_buffers('new_size')`.

This patch fits the discussion at [1]. I have responded there. Please
take a look.

[1] https://www.postgresql.org/message-id/cnthxg2eekacrejyeonuhiaezc7vd7o2uowlsbenxqfkjwgvwj%40qgzu6eoqrglb

I am attaching the resizable shared memory structures patch from the
latest patchset I have posted on thread [1]. I am posting it here
since it's easy to continue reviewing the work with prior context in
this thread. That thread may be suitable to review the API in the
context of shared buffer resizing.

I have now squashed all the changes 1. memory protection, 2.
portability, 3. min-max support into a single attached patch. Some
noticeable improvements:
1. mmap now uses MAP_NORESERVE so that physical memory for future
growth is not required upfront. The server will need initially
required memory to be present.
2. Memory protection happens outside init_fn/attach_fn for all the
resizable structures at a time at the startup.
3. Multiple GUCs to track minimum, maximum and initial shared memory
requirements.

Commit message describes the changes in detail and also lists open
items, TODOs that need more work and discussion.

--
Best Wishes,
Ashutosh Bapat