Re: Optimize planner memory consumption for huge arrays

On 2/25/24 17:29, Tom Lane wrote:
> Tomas Vondra <tomas(dot)vondra(at)enterprisedb(dot)com> writes:
>> On 2/25/24 00:07, Tom Lane wrote:
>>> ... I'm not sure if it'd be worth extending
>>> the mcxt.c API to provide something like "MemoryContextResetIfBig",
>>> with some internal rule that would be cheap to apply like "reset
>>> if we have any non-keeper blocks".
>
>> I think MemoryContextResetIfBig is an interesting idea - I think a good
>> way to define "big" would be "has multiple blocks", because that's the
>> only case where we can actually reclaim some memory.
>
> Yeah. Also: once we had such an idea, it'd be very tempting to apply
> it to other frequently-reset contexts like the executor's per-tuple
> evaluation contexts. I'm not quite prepared to argue that
> MemoryContextReset should just act that way all the time ... but
> it's sure interesting to think about.
>

Do the context resets consume enough time to make this measurable? I may
be wrong, but I'd guess it's not measurable. In which case, what would
be the benefit?

> Another question is whether this wouldn't hurt debugging, in that
> dangling-pointer bugs would become harder to catch. We'd certainly
> want to turn off the optimization in USE_VALGRIND builds, and maybe
> we just shouldn't do it at all if USE_ASSERT_CHECKING.
>
> regards, tom lane

+1 to disable this optimization in assert-enabled builds. I guess we'd
invent a new constant to disable it, and tie it to USE_ASSERT_CHECKING
(similar to CLOBBER_FREED_MEMORY, for example).

Thinking about CLOBBER_FREED_MEMORY, could it be useful to still clobber
the memory, even if we don't actually reset the context?

regards

--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company