Re: Copy data to DSA area

Hello.

At Wed, 28 Nov 2018 05:13:26 +0000, "Ideriha, Takeshi" <ideriha(dot)takeshi(at)jp(dot)fujitsu(dot)com> wrote in <4E72940DA2BF16479384A86D54D0988A6F3BD73A(at)G01JPEXMBKW04>
> Hi
>
> >From: Thomas Munro [mailto:thomas(dot)munro(at)enterprisedb(dot)com]
> >Here we need something better, because once this new kind of memory fills up with
> >leaked objects, you have to restart the whole cluster to recover.
>
> Thank you for the clarification. I agree that leaving memory leaking in shared memory
> without freeing it ends up cluster-wide problem.

If some of the "leaked" objects are liked each other with other
"live" objects including those on local memory, we must tell
whether each "maybe-leaked" object is actually a leak or
not. That looks apprently stupid. Maybe we shouldn't manipulate
objects on shared memory at such a low-level interface. We would
need a kind of resource manager for such structures holding
shared objects and should (perhaps explicitly) register the
objects that shuold be free'd on error-exiting from a scope. This
seems a bit different from reset callbacks of the current
MemoryContext.

> >If the solution involves significantly different control flows (like PG_TRY(),
> >PG_FINALLY() manual cleanup all over the place), then lots of existing palloc-based
> >code will need to be reviewed and rewritten very carefully for use in this new kind of
> >memory context. If it can be done automagically, then more of our existing code will
> >be able to participate in the construction of stuff that we want to put in shared
> >memory.
>
> About resetting X->a I was thinking about how to treat dangling pointer inside a new memory
> context machinery instead of using PG_TRY() stuff. That's because putting PG_TRY() all over the
> place becomes responsible for a developer trying to shared-memory-version MemoryContext and
> it needs a lot of efforts as you noted. But in my mind even if PG_TRY() is used, it only affects new
> code uses shared-memory version MemoryContext and doesn't affect current code.
> I may be missing something here...

Thomas mentioned exiting upper-layer library code using palloc,
like pg_list. As I wrote above, pg_list is one of such feature
that would need to be carefully adjusted.

> >I first thought about this in the context of shared plan caches, a topic that appears on
> >this mailing list from time to time. There are some other fun problems, like cache
> >invalidation for shared caches, space recycling (LRUs etc), and of course
> >locking/concurrency
> >(dsa_allocate() and dsa_free() are concurrency safe, but whatever data structures
> >you build with the memory of course need their own plan for dealing with concurrency).
> >But a strategy for clean-up on errors seems to be a major subproblem to deal with
> >early in the project. I will be very happy if we can come up with something :-)
>
> Yeah, I hope we can do it somehow.

In addition to recycling issue within a fixed area, I suppose
that we will need to "expand" the fixed-address shared
memory. PROT_NONE could work for that but it must break some
platforms..

> >On Wed, Nov 14, 2018 at 12:40 AM Ideriha, Takeshi <ideriha(dot)takeshi(at)jp(dot)fujitsu(dot)com>
> >wrote:
> >> From: Thomas Munro [mailto:thomas(dot)munro(at)enterprisedb(dot)com]
> >> >postgres=# call hoge_list(3);
> >> >NOTICE: Contents of list:
> >> >NOTICE: 1
> >> >NOTICE: 2
> >> >NOTICE: 3
> >> >CALL
> >> >
> >> >You can call that procedure from various different backends to add
> >> >and remove integers from a List. The point is that it could just as
> >> >easily be a query plan or any other palloc-based stuff in our tree, and the pointers
> >work in any backend.
> >>
> >> Thanks for the patch. I know it's a rough sketch but basically that's what I want to
> >do.
> >> I tried it and it works well.
> >
> >After your message I couldn't resist a small experiment. But it seems there are
> >plenty more problems to be solved to make it useful...
>
> Thank you for kind remark.
> By the way I just thought meta variable "hoge" is used only in Japan :)

I've seen it many-many times including mine in this list, thus
why don't you know it is already a world-wide jargon? :p

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center