Re: Why creating GIN table index is so slow than inserting data into empty table with the same index?

From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Sergey Burladyan <eshkinkot(at)gmail(dot)com>
Cc: Oleg Bartunov <oleg(at)sai(dot)msu(dot)su>, Teodor Sigaev <teodor(at)sigaev(dot)ru>, pgsql-performance(at)postgresql(dot)org
Subject: Re: Why creating GIN table index is so slow than inserting data into empty table with the same index?
Date: 2009-03-24 03:35:12
Message-ID: 18581.1237865712@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-performance

Sergey Burladyan <eshkinkot(at)gmail(dot)com> writes:
> show maintenance_work_mem ;
> maintenance_work_mem
> ----------------------
> 128MB

> create table a (i1 int, i2 int, i3 int, i4 int, i5 int, i6 int);
> insert into a select n, n, n, n, n, n from generate_series(1, 100000) as n;
> create index arr_gin on a using gin ( (array[i1, i2, i3, i4, i5, i6]) );

[ takes forever ]

Seems the reason this is so awful is that the incoming data is exactly
presorted, meaning that the tree structure that ginInsertEntry() is
trying to build degenerates to a linear list (ie, every new element
becomes the right child of the prior one). So the processing becomes
O(N^2) up till you reach maintenance_work_mem and flush the tree. With
a large setting for maintenance_work_mem it gets spectacularly slow.

I think a reasonable solution for this might be to keep an eye on
maxdepth and force a flush if that gets too large (more than a few
hundred, perhaps?). Something like this:

/* If we've maxed out our available memory, dump everything to the index */
+ /* Also dump if the tree seems to be getting too unbalanced */
- if (buildstate->accum.allocatedMemory >= maintenance_work_mem * 1024L)
+ if (buildstate->accum.allocatedMemory >= maintenance_work_mem * 1024L ||
+ buildstate->accum.maxdepth > DEPTH_LIMIT)
{

The new fast-insert code likely will need a similar defense.

Comments?

regards, tom lane

In response to

Responses

Browse pgsql-performance by date

  From Date Subject
Next Message Kouber Saparev 2009-03-24 10:52:24 Re: LIMIT confuses the planner
Previous Message Sergey Burladyan 2009-03-24 00:59:34 Why creating GIN table index is so slow than inserting data into empty table with the same index?