Re: Excessive memory usage in multi-statement queries w/ partitioning

Horiguchi-san,

Thanks for the comment. My reply is a bit late now, but....

On Wed, Jul 10, 2019 at 5:39 PM Kyotaro Horiguchi
<horikyota(dot)ntt(at)gmail(dot)com> wrote:
> At Wed, 10 Jul 2019 16:35:18 +0900, Amit Langote <amitlangote09(at)gmail(dot)com> wrote:
> - * Switch to appropriate context for constructing querytrees (again,
> - * these must outlive the execution context).
> + * Switch to appropriate context for constructing query and plan trees
> + * (again, these must outlive the execution context). Normally, it's
> + * MessageContext, but if there are more queries to plan, we use a
> + * temporary child context that will be reset after executing this
> + * query. We avoid that overhead of setting up a separate context
> + * for the common case of having just a single query.
>
> Might be stupid, but I feel uneasy that "usually it must live in
> MessageContxt, but not necessarily if there is succeeding
> query".. *I* need more explanation why it is safe to use that
> short-lived context.

So the problem we're trying solve with this is that memory consumed
when parsing/planning individual statements pile up in a single
context (currently, MessageContext), which can lead to severe bloat
especially if the planning of individual statements consumes huge
amount of memory. The solution is to use a sub-context of
MessageContext for each statement that's reset when its execution is
finished. I think it's safe to use a shorter-lived context for each
statement because the memory of a given statement should not need to
be referenced when its execution finishes. Do you see any problem
with that assumption?

Thanks,
Amit