Re: Subquery pull-up increases jointree search space

On 6/9/26 09:08, Andrei Lepikhov wrote:
> On 08/06/2026 17:27, Tomas Vondra wrote:
>> On 6/8/26 15:32, Andrei Lepikhov wrote:
>>> Instead, I used the heaptrack tool, which gave me the following
>> Hmm, how does heaptrack deal with out memory pools? I was worried
>> existing memory profilers (like heaptrack) would get confused by our
>> memory contexts, attributing the whole block to the palloc that just
>> happens to allocate a new block. But that's not really right.
>
> In theory, heaptrack does not catch pallocs inside a memory context and only
> tracks malloc and free calls. However, in my experience, it still helps identify
> the issue. This might be because the planner uses local memory contexts, and
> with [1] added, it could be even more localised. Usually, the problem is a cycle
> that grows too long, with many allocations within it that go unreleased, causing
> overconsumption and memory context growth. In this case, heaptrack may not point
> to the exact spot, but it still provides useful hints.
>

I don't know what you mean by "planner uses local memory contexts". All
the memory allocations I discussed are in PortalContext. Maybe there are
some short-lived memory contexts, but those are by definition not an
issue, because the memory is released. It's about the long-lived
allocations that are left in PortalContext that ...

Maybe heaptrack can still do a sufficient job, because given enough
allocations the distribution of palloc and malloc will be about the
same. But if there's a problem to make it work well with pools ...

> Compared to other profilers (gperftools, jemalloc, valgrind massif), I like it
> more for its flexibility, flamegraphs, and an interface that the support team
> frequently uses (even on Windows somehow) - it is beneficial for the
> inter-department communication. Profilers don’t understand malloc/palloc
> postgres separation anyway, and to use them we need to annotate postgres calls.
>

OK

The one advantage of massif might be that we already have dependency and
instrumentation for valgrind (I'm assuming it might extend that, not
having to introduce a completely new dependency).

>>
>> I see the heaptrack README claims it can work with memory pools after
>> annotating the code in some way. But there's not much details about
>> that. Also, it suggests valgrind/massif can already do that.
>>
>
> Looking into the interface [2,3], it looks quite clear to me - it even includes
> a Valgrind macros set that makes it more natively integrated. Thinking about
> implementation, my first wish was to create an extension and see how it actually
> works in practice, but we can’t inject a wrapper into the static
> MemoryContextMethods, which limits the ability of such an extension. Maybe with
> an umbrella planning memory context …
>
> Core-based annotation looks more interesting. There are questions about proper
> palloc/malloc counting that exist - I think it can be generally solved by
> employing heaptrack_pause / heaptrack_resume calls, if this topic sparks an
> interest. Or we can discuss some API flexibility on the HeapTrack side.
>

I don't see a point of doing this from an extension. It's a developer
feature, similar to valgrind instrumentation.

regards

--
Tomas Vondra