RE: Re: Spilling hashed SetOps and aggregates to disk

Strange. We don't see this overeahd and measure a lot more than just a single metric:

Of the grab below only the context->stats += size is need.
Could easily be a macro.
We also track overall backend size to cap it, and track memory contexts in shared memory (plan cache, function cache, message and transaction contexts).
Each with high watermarks.

/*
* NOTE: Shared contexts stats are global and shared by
* all engines, so, never add the size directly, instead
* use stats API which uses Atomic.*() calls to ensure
* mutual exclusion.
*/
static void MaintainAllocStats(MemoryContext context, Size size)
{
if (context->stats)
{
if (!context->isShared)
{
sMonAdd(memory_context_stats, TotalBackendAllocated_Size, size);
sMonHWM(memory_context_stats, TotalBackendAllocated_HWM,
sMonGet(memory_context_stats, TotalBackendAllocated_Size));
context->allocatedSize += size;
context->stats[0] += size;
context->stats[1] = (context->stats[1] >= context->stats[0] ?
context->stats[1] : context->stats[0]);
}
else
{
if (!context->isNested)
size += SHM_BLOCK_OVERHEAD;

sAtomicAdd(&context->allocatedSize, size);

if (!context->isNested ||
!context->clusterrep ||
context->clusterrep->stats != context->stats)
sAtomicAdd(context->stats, size);
}
}
}

I'll try myself on peeling out a patch for community for the stats and ask he developer responsible for hash spilling to engage.

Cheers
Serge

--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services