RE: speeding up planning with partitions

From: "Tsunakawa, Takayuki" <tsunakawa(dot)takay(at)jp(dot)fujitsu(dot)com>
To: "Imai, Yoshikazu" <imai(dot)yoshikazu(at)jp(dot)fujitsu(dot)com>
Cc: 'Amit Langote' <Langote_Amit_f8(at)lab(dot)ntt(dot)co(dot)jp>, David Rowley <david(dot)rowley(at)2ndquadrant(dot)com>, Alvaro Herrera <alvherre(at)2ndquadrant(dot)com>, "Pg Hackers" <pgsql-hackers(at)postgresql(dot)org>
Subject: RE: speeding up planning with partitions
Date: 2019-02-06 02:03:34
Message-ID: 0A3221C70F24FB45833433255569204D1FB9423C@G01JPEXMBYT05
Views: Raw Message | Whole Thread | Download mbox
Thread:
Lists: pgsql-hackers

Imai-san,

From: Amit Langote [mailto:Langote_Amit_f8(at)lab(dot)ntt(dot)co(dot)jp]
> On 2019/01/22 18:47, David Rowley wrote:
> > On Tue, 22 Jan 2019 at 20:01, Imai, Yoshikazu
> >> What I understand so far is about 10,000 while loops at total
> (4098+4098+some extra) is needed in hash_seq_search() in EXECUTE query
> after the creation of the generic plan.
> >> 10,000 while loops takes about 10 microsec (of course, we can't estimate
> correct time), and the difference of the latency between 5th and 7th EXECUTE
> is about 8 microsec, I currently think this causes the difference.
> > >
> >> I don't know this problem relates to Amit-san's patch, but I'll continue
> to investigate it.
> >
> > I had another thought... when you're making a custom plan you're only
> > grabbing locks on partitions that were not pruned (just 1 partition in
> > your case), but when making the generic plan, locks will be acquired
> > on all partitions (all 4000 of them). This likely means that when
> > building the generic plan for the first time that the
> > LockMethodLocalHash table is expanded to fit all those locks, and
> > since we never shrink those down again, it'll remain that size for the
> > rest of your run. I imagine the reason for the slowdown is that
> > during LockReleaseAll(), a sequential scan is performed over the
> > entire hash table. I see from looking at the hash_seq_search() code
> > that the value of max_bucket is pretty critical to how it'll perform.
> > The while ((curElem = segp[segment_ndx]) == NULL) loop will need to
> > run fewer times with a lower max_bucket.
>
> I too think that that might be behind that slight drop in performance.
> So, it's good to know what one of the performance bottlenecks is when
> dealing with large number of relations in queries.

Can you compare the performance of auto and force_custom_plan again with the attached patch? It uses PGPROC's LOCALLOCK list instead of the hash table.

Regards
Takayuki Tsunakawa

Attachment Content-Type Size
faster-locallock-scan.patch application/octet-stream 6.7 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Alvaro Herrera 2019-02-06 02:10:37 Re: Undo logs
Previous Message Alvaro Herrera 2019-02-06 02:02:20 Re: bug tracking system