Re: gin fast insert performance

> Here is a test of the fast insert patch. The patch has gone through some
> changes, so this set of tests is to see the current performance impact
> compared with HEAD.
>
> The test is simple: inserting a bunch of integer arrays into a table
> with a GIN index on the array column.
>
> I'm testing with small work_mem and large work_mem because the smaller
> work mem should be periodically flushing the pending list throughout a
> large insert, while large work_mem should allow larger batches to build
> up before flushing the pending list. For HEAD, larger work_mem should
> have no effect.

You didn't provide distributions of array's element, number of unique element
and so on. And I make simple test script, which generates data rather close to
typical tsearch installation (see tst.sql).

And results on my notebook (in seconds):
fastupdate | insert | vacuum
------------+------------+--------
100000 rows
off | 316.147 | 0.770
on | 65.461 | 12.795
1000000 rows
off | >16 hours | -
on | 6612.595 | 12.795

I stop the test with fastupdate=off and one million rows - it ran too long :).
Changes in postgresql.conf:
shared_buffers=128MB
temp_buffers=16MB
work_mem=16MB
maintenance_work_mem=256MB
effective_cache_size=1024MB
autovacuum=off

Fastest way is create table, fill it, create index and vacuum it (for 100000
records):
17 secs to insert
27 secs to create an index
1 second to vacuum
So, in summary, it takes 45 secs instead of 78 secs with fast update and 317
seconds without fast update. I think, it's a win in performance.

> With the fast insert patch, the total time for insert + vacuum isn't
> much different with increased work_mem, but increased work_mem clearly
> defers a lot of the work to VACUUM.

"but increased work_mem clearly *may* defer a lot of the work to VACUUM."
Because in real world it's impossible to predict when clearing of pending list
will be started. And autovacuum usually will fire the clearing earlier than
pending list reaches the limit.

> So, unfortunately, the fast insert patch does not appear to bring the
> overall time anywhere close to building the index from scratch. When the
> work_mem is set to 1GB, the VACUUM took about twice as long to run than
> the entire index build. Teodor, can you explain why that might be?
Yeah, creation of index is much more optimizable than sequential insertions.
With enabled fast update there is a overhead of writing pending pages (and WAL
too), and that pages should be readed to collect data into memory. Next,
clearing process uses work_mem instead of maintenance_work_mem, which is usually
greater. Algorithm of bulk insertion (it's used in creation and cleaning too)
likes tids at the end of table, if lowest tid to insert is greater than lastest
tid in current tree then algorithm could insert more than one tid at once.

> It does show improvement, and I think my test case might just be too
If dataset is bigger then improvement is better :).

> small. It seems like a lot of time is used inserting into the pending
> list, but it seems like it should be a simple operation. Maybe that can
> be optimized?

As you can see (ginfast.c), insertion into pending list could cause not fully
filled pages, in worst case pending list will contain about 50% of unused space
(if every indexed value takes GIN_PAGE_FREESIZE+1 bytes then value will takes
two pages). This is a price to keep concurrency at high level :(. If you have an
idea how to do compact, fast and concurrent insertion into pending list (or
another structure) and keep reasonable time to search, please, don't be quiet :)

--
Teodor Sigaev E-mail: teodor(at)sigaev(dot)ru
WWW: http://www.sigaev.ru/