Re: [report] memory leaks in COPY FROM on partitioned table

On 2018-Aug-06, Kohei KaiGai wrote:

> 2018-08-06 1:50 GMT+09:00 Alvaro Herrera <alvherre(at)2ndquadrant(dot)com>:
> >> Now, it consumed about 60MB rss at the beginning of COPY FROM, and it
> >> grows up very slowly during the COPY FROM execution, then grew up to
> >> 250MB before completion.
> >> We may have another memory blocks which are not released during
> >> execution, however,
> >> I could not identify whether it is really memory leaking or not, and
> >> who's jobs.
> >
> > Most likely, this is a different memory leak.
> >
> > I sugges that one way to track this down is first figure out *which*
> > context is the one growing, which you can see by running
> > MemoryContextStats a few times and noting for meaningful differences.
> > Then we can try to narrow down what is allocating stuff in that context.
> >
> Yes, but the hardest is identification of which memory context is growing
> up time by time. Once we could identify, MemoryContextStats() tells us
> the name of problematic context and details.

Well, I was thinking you'd call MemCxtStats on TopMemoryContext and
observe changes in the whole hierarchy. However ...

> Of course, above my observation is just based on rss usage of postgresql.
> It can increase physical page allocation by page fault on the virtual address
> space correctly allocated.

... this is a good point too, and I'm not sure to what extent this
problem is fixable.

> >> It may be an idea to put a debug code that raises a notice when
> >> MemoryContext allocates more than the threshold.
> >
> > I don't think this is really practical, because whatever the threshold
> > we set, there would be some corner-case scenario where the threshold is
> > legitimately crossed. And some memory leak scenarios that don't cross
> > any thresholds.
> >
> I assume this threshold is configurable by GUC, and disabled on the default.
> Once a user found suspicious memory usage increase, we can set a threshold
> value. In above case, we may be able to see something around 120MB threshold.

Okay. I suppose you'd want to improve traceability of allocations in
some more general way, but I think I understand your point about the
threshold. Seems overly specific, but maybe it's okay.

--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services