Re: OOM on EXPLAIN with lots of nodes

From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Heikki Linnakangas <hlinnakangas(at)vmware(dot)com>
Cc: Alvaro Herrera <alvherre(at)2ndquadrant(dot)com>, Alexey Bashtanov <bashtanov(at)imap(dot)cc>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: OOM on EXPLAIN with lots of nodes
Date: 2015-01-14 01:16:33
Message-ID: 3779.1421198193@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

I wrote:
> Heikki Linnakangas <hlinnakangas(at)vmware(dot)com> writes:
>> But do we really need to backpatch any of this?

> Alexey's example consumes only a couple hundred MB in 9.2, vs about 7GB
> peak in 9.3 and up. That seems like a pretty nasty regression.

I did a bit more measurement of the time and backend memory consumption
for Alexey's example EXPLAIN:

9.2: 0.9 sec, circa 200 MB
HEAD: 56 sec, circa 7300 MB
with patch below: 3.7 sec, circa 300 MB

So while this doesn't get us all the way back down to where we were before
we started trying to guarantee unique table/column identifiers in EXPLAIN
printouts, it's at least a lot closer.

Not sure whether to just commit this to HEAD and call it a day, or to
risk back-patching.

It occurred to me that we could use your idea of a secondary data
structure and minimize the code impact with a macro layer, ie
#define grouping_stack pointer_field->groupingstack
But I'm not sure if it's worth the trouble. 9.3 has been like this
right along, and this is the first complaint.

One compromise idea would be to back-patch only as far as 9.4.
If there are extensions out there that have this ABI issue, expecting
them to rebuild for 9.4.1 might be OK.

regards, tom lane

Attachment Content-Type Size
explain-memory-consumption-fix.patch text/x-diff 8.8 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Peter Geoghegan 2015-01-14 01:24:03 Re: hung backends stuck in spinlock heavy endless loop
Previous Message Peter Geoghegan 2015-01-14 00:47:56 Re: hung backends stuck in spinlock heavy endless loop