Make MemoryContextMemAllocated() more precise

From: Jeff Davis <pgsql(at)j-davis(dot)com>
To: pgsql-hackers(at)postgresql(dot)org
Cc: Tomas Vondra <tomas(dot)vondra(at)postgresql(dot)org>
Subject: Make MemoryContextMemAllocated() more precise
Date: 2020-03-16 18:45:14
Message-ID: ec63d70b668818255486a83ffadc3aec492c1f57.camel@j-davis.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers


AllocSet allocates memory for itself in blocks, which double in size up
to maxBlockSize. So, the current block (the last one malloc'd) may
represent half of the total memory allocated for the context itself.

The free space at the end of that block hasn't been touched at all, and
doesn't represent fragmentation or overhead. That means that the
"allocated" memory can be 2X the memory ever touched in the worst case.

Although that's technically correct, the purpose of
MemoryContextMemAllocated() is to give a "real" usage number so we know
when we're out of work_mem and need to spill (in particular, the disk-
based HashAgg work, but ideally other operators as well). This "real"
number should include fragmentation, freed-and-not-reused chunks, and
other overhead. But it should not include significant amounts of
allocated-but-never-touched memory, which says more about economizing
calls to malloc than it does about the algorithm's memory usage.

Attached is a patch that makes mem_allocated a method (rather than a
field) of MemoryContext, and allows each memory context type to track
the memory its own way. They all do the same thing as before
(increment/decrement a field), but AllocSet also subtracts out the free
space in the current block. For Slab and Generation, we could do
something similar, but it's not as much of a problem because there's no
doubling of the allocation size.

Although I think this still matches the word "allocation" in spirit,
it's not technically correct, so feel free to suggest a new name for
MemoryContextMemAllocated().

Regards,
Jeff Davis

Attachment Content-Type Size
allocated.patch text/x-patch 13.6 KB

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Andres Freund 2020-03-16 18:57:02 Re: Berserk Autovacuum (let's save next Mandrill)
Previous Message Fabien COELHO 2020-03-16 18:21:06 Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)