Re: Optimize planner memory consumption for huge arrays

From: Tomas Vondra <tomas(dot)vondra(at)enterprisedb(dot)com>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: Lepikhov Andrei <a(dot)lepikhov(at)postgrespro(dot)ru>, Ashutosh Bapat <ashutosh(dot)bapat(dot)oss(at)gmail(dot)com>, pgsql-hackers(at)lists(dot)postgresql(dot)org, Евгений Бредня <e(dot)brednya(at)postgrespro(dot)ru>
Subject: Re: Optimize planner memory consumption for huge arrays
Date: 2024-02-26 09:01:31
Message-ID: b34b5fa6-2f35-4b49-823d-94e61aa0368d@enterprisedb.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On 2/25/24 17:29, Tom Lane wrote:
> Tomas Vondra <tomas(dot)vondra(at)enterprisedb(dot)com> writes:
>> On 2/25/24 00:07, Tom Lane wrote:
>>> ... I'm not sure if it'd be worth extending
>>> the mcxt.c API to provide something like "MemoryContextResetIfBig",
>>> with some internal rule that would be cheap to apply like "reset
>>> if we have any non-keeper blocks".
>
>> I think MemoryContextResetIfBig is an interesting idea - I think a good
>> way to define "big" would be "has multiple blocks", because that's the
>> only case where we can actually reclaim some memory.
>
> Yeah. Also: once we had such an idea, it'd be very tempting to apply
> it to other frequently-reset contexts like the executor's per-tuple
> evaluation contexts. I'm not quite prepared to argue that
> MemoryContextReset should just act that way all the time ... but
> it's sure interesting to think about.
>

Do the context resets consume enough time to make this measurable? I may
be wrong, but I'd guess it's not measurable. In which case, what would
be the benefit?

> Another question is whether this wouldn't hurt debugging, in that
> dangling-pointer bugs would become harder to catch. We'd certainly
> want to turn off the optimization in USE_VALGRIND builds, and maybe
> we just shouldn't do it at all if USE_ASSERT_CHECKING.
>
> regards, tom lane

+1 to disable this optimization in assert-enabled builds. I guess we'd
invent a new constant to disable it, and tie it to USE_ASSERT_CHECKING
(similar to CLOBBER_FREED_MEMORY, for example).

Thinking about CLOBBER_FREED_MEMORY, could it be useful to still clobber
the memory, even if we don't actually reset the context?

regards

--
Tomas Vondra
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Andrey M. Borodin 2024-02-26 09:10:49 Re: Injection points: some tools to wait and wake
Previous Message Japin Li 2024-02-26 09:00:13 Re: Improve readability by using designated initializers when possible