Skip site navigation (1) Skip section navigation (2)

Re: rbtree code breaks GIN's adherence to maintenance_work_mem

From: Robert Haas <robertmhaas(at)gmail(dot)com>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: pgsql-hackers(at)postgresql(dot)org
Subject: Re: rbtree code breaks GIN's adherence to maintenance_work_mem
Date: 2010-07-31 12:42:44
Message-ID: AANLkTinXYxKWK_-bfH+05dZWhY286P3SxzwQ_5V4Tg5f@mail.gmail.com (view raw or flat)
Thread:
Lists: pgsql-hackers
On Sat, Jul 31, 2010 at 12:40 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Another misbehavior that I noted while playing with Artur's example is
> that, while GIN index build seems to adhere pretty well to whatever
> maintenance_work_mem limit you give it in 8.4, it blows right by that
> limit in 9.0 and HEAD --- I observed it happily eating something north
> of 128MB with a maintenance_work_mem of 70MB.  It looks to me like the
> reason is the new "rbtree.c" code, which adds a whole lot of data
> structure that's not being counted against the maintenance_work_mem
> limit.
>
> Now the first question that might be asked is what we'd need to do to
> rbtree.c's API to allow its memory consumption to be tracked.  However,
> I think there's a considerably more pressing question, which is what
> is it that rbtree.c is doing that justifies a 2X bloat factor in GIN
> index build --- which is pretty much what it's costing us, given the
> observation above.  We know that the amount of memory available for
> the build has an enormous effect on build time.  If we just do the
> obvious thing of including the rbtree data structure in the
> maintenance_work_mem calculation, what we're going to get is a very
> substantial slowdown in build speed for the same maintenance_work_mem
> setting, compared to the way 8.4 worked.
>
> So, I would like somebody to show cause why that whole module shouldn't
> be ripped out and the code reverted to where it was in 8.4.  My
> recollection is that the argument for adding it was to speed things up
> in corner cases, but what I think it's actually going to do is slow
> things down in every case.

I've always been a bit suspicious of this code, too, even though I
didn't think about the memory consumption issue.  But see here:

http://archives.postgresql.org/pgsql-hackers/2010-02/msg00307.php

I think there may be a better way to avoid the pathological cases, but
I'm not sure what it is.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

In response to

Responses

pgsql-hackers by date

Next:From: Tom LaneDate: 2010-07-31 13:57:13
Subject: More fun with GIN lossy-page pointers
Previous:From: Peter EisentrautDate: 2010-07-31 12:10:33
Subject: Re: review: xml_is_well_formed

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group