Optimize planner memory consumption for huge arrays

From: "Lepikhov Andrei" <a(dot)lepikhov(at)postgrespro(dot)ru>
To: pgsql-hackers(at)lists(dot)postgresql(dot)org
Cc: Евгений Бредня <e(dot)brednya(at)postgrespro(dot)ru>
Subject: Optimize planner memory consumption for huge arrays
Date: 2023-09-04 05:25:44
Message-ID: em9939439a-441a-4b27-a977-ebdf5987dc49@7d14f008.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hi, hackers,

Looking at the planner behaviour with the memory consumption patch [1], I figured out that arrays increase memory consumption by the optimizer significantly. See init.sql in attachment.
The point here is that the planner does small memory allocations for each element during estimation. As a result, it looks like the planner consumes about 250 bytes for each integer element.

It is maybe not a problem most of the time. However, in the case of partitions, memory consumption multiplies by each partition. Such a corner case looks weird, but the fix is simple. So, why not?

The diff in the attachment is proof of concept showing how to reduce wasting of memory. Having benchmarked a bit, I didn't find any overhead.

[1] Report planning memory in EXPLAIN ANALYZE
https://www.postgresql.org/message-id/flat/CAExHW5sZA=5LJ_ZPpRO-w09ck8z9p7eaYAqq3Ks9GDfhrxeWBw(at)mail(dot)gmail(dot)com

--
Regards,
Andrey Lepikhov

Attachment Content-Type Size
init.sql application/sql 344 bytes
array_compact_memusage.diff application/octet-stream 1.5 KB

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Amit Kapila 2023-09-04 05:48:24 Re: Impact of checkpointer during pg_upgrade
Previous Message Dilip Kumar 2023-09-04 05:03:20 Re: Impact of checkpointer during pg_upgrade