From: | "Sean Davis" <sdavis2(at)mail(dot)nih(dot)gov> |
---|---|
To: | "Tom Lane" <tgl(at)sss(dot)pgh(dot)pa(dot)us> |
Cc: | pgsql-novice(at)postgresql(dot)org |
Subject: | Re: Building full-text index |
Date: | 2007-11-16 22:28:03 |
Message-ID: | 264855a00711161428j62d510cci59882f6331b0b0ce@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-novice |
On Nov 16, 2007 5:00 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> "Sean Davis" <sdavis2(at)mail(dot)nih(dot)gov> writes:
> > I am trying to build a full-text index (gin(to_tsvector('english',
> > title || abstract))) on about 18 million abstracts and titles from
> > medical literature. However, I keep getting out-of-memory errors. (I
> > am on a 32Gb linux system with maintenance_work_mem set to 20Gb and
> > shared buffers at 4Gb; postgres 8.3beta). Does creation of a
> > full-text index require that the entire index fit into memory?
>
> I looked closer at this and discovered that there's an overflow problem
> in the GIN index build code: with maintenance_work_mem above 8Gb, it
> miscalculates how much space it's used and never realizes when it's
> reached the intended limit. So indeed you were seeing it try to create
> the entire index in memory :-(.
>
> This will be fixed in the next beta, but in the meantime set
> maintenance_work_mem to something less than 8Gb.
Thanks, Tom. I had tried this empirically and things worked fine.
Glad to hear that it is fixed in the next beta.
Sean
From | Date | Subject | |
---|---|---|---|
Next Message | Mary Anderson | 2007-11-16 23:07:09 | Accessing tables in one database from another |
Previous Message | Tom Lane | 2007-11-16 22:00:36 | Re: Building full-text index |