Re: rbtree code breaks GIN's adherence to maintenance_work_mem

On Sat, Jul 31, 2010 at 12:40 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Another misbehavior that I noted while playing with Artur's example is
> that, while GIN index build seems to adhere pretty well to whatever
> maintenance_work_mem limit you give it in 8.4, it blows right by that
> limit in 9.0 and HEAD --- I observed it happily eating something north
> of 128MB with a maintenance_work_mem of 70MB.  It looks to me like the
> reason is the new "rbtree.c" code, which adds a whole lot of data
> structure that's not being counted against the maintenance_work_mem
> limit.
>
> Now the first question that might be asked is what we'd need to do to
> rbtree.c's API to allow its memory consumption to be tracked.  However,
> I think there's a considerably more pressing question, which is what
> is it that rbtree.c is doing that justifies a 2X bloat factor in GIN
> index build --- which is pretty much what it's costing us, given the
> observation above.  We know that the amount of memory available for
> the build has an enormous effect on build time.  If we just do the
> obvious thing of including the rbtree data structure in the
> maintenance_work_mem calculation, what we're going to get is a very
> substantial slowdown in build speed for the same maintenance_work_mem
> setting, compared to the way 8.4 worked.
>
> So, I would like somebody to show cause why that whole module shouldn't
> be ripped out and the code reverted to where it was in 8.4.  My
> recollection is that the argument for adding it was to speed things up
> in corner cases, but what I think it's actually going to do is slow
> things down in every case.

I've always been a bit suspicious of this code, too, even though I
didn't think about the memory consumption issue. But see here:

http://archives.postgresql.org/pgsql-hackers/2010-02/msg00307.php

I think there may be a better way to avoid the pathological cases, but
I'm not sure what it is.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company