Re: excessive amounts of consumed memory (RSS), triggering OOM killer

On 2.12.2014 02:52, Tom Lane wrote:
> Tomas Vondra <tv(at)fuzzy(dot)cz> writes:
>> On 2.12.2014 01:33, Tom Lane wrote:
>>> What I suspect you're looking at here is the detritus of creation
>>> of a huge number of memory contexts. mcxt.c keeps its own state
>>> about existing contents in TopMemoryContext. So, if we posit that
>>> those contexts weren't real small, there's certainly room to
>>> believe that there was some major memory bloat going on recently.
>
>> Aha! MemoryContextCreate allocates the memory for the new context
>> from TopMemoryContext explicitly, so that it survives resets of the
>> parent context. Is that what you had in mind by keeping state about
>> existing contexts?
>
> Right.
>
>> That'd probably explain the TopMemoryContext size, because
>> array_agg() creates separate context for each group. So if you have
>> 1M groups, you have 1M contexts. Although I don's see how the size
>> of those contexts would matter?
>
> Well, if they're each 6K, that's your 6GB right there.

FWIW, I tried to repeat the test with the array_agg() patch, submitted
to the next CF:

http://www.postgresql.org/message-id/5479FAEA.3070501@fuzzy.cz

and the memory context stats after the first iteration look like this:

master (unpatched)
------------------
TopMemoryContext: 204878128 total in 25011 blocks; 204010192 free
(750034 chunks); 867936 used

patched with array_agg
----------------------
TopMemoryContext: 78128 total in 11 blocks; 10144 free (34 chunks);
67984 used

Also, the RSS issue (which was the initial subject of this thread)
pretty much disappeared. So indeed, this seems to be caused by the
'islands' referenced by TopMemoryContext.

I wonder whether it'd be worth looking for places doing something
similar to array_agg(), potentially creating many contexts in parallel,
and maybe modifying them as in the patch.

regards
Tomas