Re: dynamically allocating chunks from shared memory

On Mon, Aug 9, 2010 at 11:41 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>> So imagine that thread-or-process A allocates allocates a new chunk of
>> memory and then writes a pointer to the new chunk in a previously
>> allocated section of memory.  Thread-or-process B then follows the
>> pointer.  In a threaded model, this is guaranteed to be safe.  In a
>> process model, it's not: A might have enlarged the shared memory
>> mapping while B has not yet done so.  So I think in our model any sort
>> of change to the shared memory segment is going to require extremely
>> careful gymnastics, and be pretty expensive.
>
> ... and on some platforms, it'll be flat out impossible.  We looked at
> this years ago and concluded that changing the size of the shmem segment
> after postmaster start was impractical from a portability standpoint.
> I have not seen anything to change that conclusion.

I haven't done extensive research into this, but I did take a look at
it briefly. It looked to me like the style of shared memory we're
using now (I guess it's System V) has no way to resize a shared memory
segment at all, and certainly no way that's portable. However it also
looked as though POSIX shm (shm_open, etc.) can be resized using
ftruncate(). Whether this is portable to all the platforms we run on,
or whether the behavior of ftruncate() in combination with shm_open()
is in the standard, I'm not sure. I believe I went back and reread
the old threads on this topic and it seems like the sticking point as
far as POSIX shm goes it that it lacks a readable equivalent of
shm_nattch. I think it was proposed to use a small syv shm and then
do the main shared memory arena with shm_open, but at that point you
start to wonder you're messing around with at all.

But I can't help but be intrigued by it, even so. Suppose, for
example, that we kept things that were really fixed-size in shared
memory but moved, say, shared_buffers to a POSIX shm. Would that
allow you to then make shared_buffers PGC_SIGHUP? The obvious answer
is "no", because there are a whole bunch of knock-on issues. Changing
the size of shared_buffers also means changing the number of LWLocks,
changing the number of buffer descriptors, etc. So maybe it can't be
done. But I can't stop wondering if there's a way to make it work...

>> I don't care to take a position in the religious war over threads vs.
>> processes, but I do think threads simplify the handling of this
>> particular case.
>
> You meant "I don't think", right?  I agree.  The only way threads would
> simplify this is if we went over to a mysql-style model where there was
> only one process, period, and all backends were threads inside that.
> No shared memory as such, at all.

I think we're saying the same thing in different ways; I agree with
everything in that paragraph that follows the question mark. By "this
particular case", I meant "shared memory allocation"; it would amount
to just calling malloc() [or palloc()]. But yeah, clearly that only
works in a single-process model.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company