Re: Reducing relcache memory usage: deduping index shapes

Hi Andres!

>> I hadn't realized that we're creating tons of memory contexts in
>> RelationData, e.g. one per index. Beyond wasting memory this is also
>> cluttering pg_backend_memory_contexts considerably.
>>
>> I had a quick look if we can easily get rid of rd_indexcxt but there's
>> some memory from the function manager that is not directly referenced
>> and can only be implicitly freed by destroying rd_indexcxt.
>
> I think that'd be the wrong direction. We shouldn't use fewer contexts, we
> should use vastly *more* contexts for the various caches. It's a real issue
> that today we

Agreed that having more granular contexts would be valuable for
debugging and attribution. I was looking at it from a different angle in
that specific case (reducing overhead) as it seemed that allocating
multiple memory contexts for an object that is possibly allocated
hundreds of thousands of times won't fly.

So we either need to change how relcache is structured or work towards
reducing the memory footprint of the relevant memory allocators. Your
proxy context might be a first step but I'm concerned it will quickly be
overused in places where we should use another context type but it's
simpler to use the proxy context than to redesign.

> a) can have leaks in CacheMemoryContext that cannot be attributed to anything
>
> b) can have huge memory usage in CacheMemoryContext - without leaks - without
> the ability to figure out why CacheMemoryContext is that big.

Seems like for these requirements we actually don't need full-fledged
memory contexts but rather something that allows labeling allocations
inside the same memory context.
> I don't think cluttering pg_backend_memory_contexts is an argument, that needs
> to be addressed on the querying side. With the path support added a few
> releases back it's not that hard anymore.
We could aggregate memory contexts with the same name and only output
them once. But we possibly need some way to drill down and look at each
context individually as well.

--
David Geier