Re: Building full-text index

On Nov 16, 2007 5:00 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> "Sean Davis" <sdavis2(at)mail(dot)nih(dot)gov> writes:
> > I am trying to build a full-text index (gin(to_tsvector('english',
> > title || abstract))) on about 18 million abstracts and titles from
> > medical literature. However, I keep getting out-of-memory errors. (I
> > am on a 32Gb linux system with maintenance_work_mem set to 20Gb and
> > shared buffers at 4Gb; postgres 8.3beta). Does creation of a
> > full-text index require that the entire index fit into memory?
>
> I looked closer at this and discovered that there's an overflow problem
> in the GIN index build code: with maintenance_work_mem above 8Gb, it
> miscalculates how much space it's used and never realizes when it's
> reached the intended limit. So indeed you were seeing it try to create
> the entire index in memory :-(.
>
> This will be fixed in the next beta, but in the meantime set
> maintenance_work_mem to something less than 8Gb.

Thanks, Tom. I had tried this empirically and things worked fine.
Glad to hear that it is fixed in the next beta.

Sean