Re: Subquery pull-up increases jointree search space

On 6/8/26 17:27, Tomas Vondra wrote:
> ...
>
> Hmm, how does heaptrack deal with out memory pools? I was worried
> existing memory profilers (like heaptrack) would get confused by our
> memory contexts, attributing the whole block to the palloc that just
> happens to allocate a new block. But that's not really right.
>
> I see the heaptrack README claims it can work with memory pools after
> annotating the code in some way. But there's not much details about
> that. Also, it suggests valgrind/massif can already do that.
>

I took a look at massif - mostly because it comes with valgrind, which
we already use and have the infrastructure in place for (while heaptrack
would be a new dependency, would require new instrumentation etc.).

I think it works pretty well. I'm still not quite sure it does the right
thing with memory contexts (i.e. tracking chunks and not malloc blocks).
We do have the macros (VALGRIND_MEMPOOL_ALLOC, ...) in the right places,
but I saw some malloc() calls in the traces too, which seems wrong.

Anyway, it seems very convenient to use, and gives me data roughly in
line with the manual tracing. It produces much nicer / more complete
reports, and it's way faster (a minute vs. an hour).

Here's how I use it (after building with -DUSE_VALGRIND)

valgrind --trace-children=yes --tool=massif \
--alloc-fn=MemoryContextAlloc \
--alloc-fn=MemoryContextAllocExtended \
--alloc-fn=AllocSetAlloc \
--alloc-fn=AllocSetAllocFromNewBlock \
--alloc-fn=palloc --alloc-fn=palloc0 \
--max-snapshots=1000 --detailed-freq=1 --threshold=0.0 \
postgres ...

Which may be an overkill (e.g. too many detailed snapshots, threshold
too low, ...). But it works. It produces one file per process, I think
you need to exit the process for the file to be written / complete.

Then, to process the massif.out.PID file into a text report:

TMPDIR=./tmp ms_print massif.out.PID > massif.txt

Attached is an example of one of the reports.

It's also possible to use a massif-visualizer, which shows a nice graph
and allows drilling down to the various places allocating memory.

regards

--
Tomas Vondra