Re: (full) Memory context dump considered harmful

On 08/21/2015 08:37 PM, Tom Lane wrote:
> Tomas Vondra <tomas(dot)vondra(at)2ndquadrant(dot)com> writes:
>>
>> I also don't think logging just subset of the stats is a lost case.
>> Sure, we can't know which of the lines are important, but for example
>> logging just the top-level contexts with a summary of the child contexts
>> would be OK.
>
> I thought a bit more about this. Generally, what you want to know about
> a given situation is which contexts have a whole lot of allocations
> and/or a whole lot of child contexts. What you suggest above won't work
> very well if the problem is buried more than about two levels down in
> the context tree. But suppose we add a parameter to memory context stats
> collection that is the maximum number of child contexts to print *per
> parent context*. If there are more than that, summarize the rest as per
> your suggestion. So any given recursion level might look like
>
> FooContext: m total in n blocks ...
> ChildContext1: m total in n blocks ...
> possible grandchildren...
> ChildContext2: m total in n blocks ...
> possible grandchildren...
> ChildContext3: m total in n blocks ...
> possible grandchildren...
> k more child contexts containing m total in n blocks ...
>
> This would require a fixed amount of extra state per recursion level,
> so it could be done with a few more parameters/local variables in
> MemoryContextStats and no need to risk a malloc().
>
> The case where you would lose important data is where the serious
> bloat is in some specific child context that is after the first N
> children of its direct parent. I don't believe I've ever seen a case
> where that was critical information as long as N isn't too tiny.

Couldn't we make it a bit smarter to handle even cases like this? For
example we might first count/sum the child contexts, and then print
either all child contexts (if there are only a few of them) or just
those that are >5% of the total, 2x the average or something like that.

regards

--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services