From: | "Magnus Hagander" <mha(at)sollentuna(dot)net> |
---|---|
To: | "Tatsuo Ishii" <ishii(at)sraoss(dot)co(dot)jp>, <teodor(at)sigaev(dot)ru> |
Cc: | <pgsql-hackers(at)postgresql(dot)org> |
Subject: | Re: string_to_array eats too much memory? |
Date: | 2006-11-08 09:47:29 |
Message-ID: | 6BCB9D8A16AC4241919521715F4D8BCEA358BA@algol.sollentuna.se |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-hackers |
> > > I'm playing with GIN to make a full text search system. GIN comes
> > > with built-in TEXT[] support and I use string_to_array()
> to make a
> > > TEXT[]. Problem is, if there's large number of array elemets,
> > > string_to_array() consumes too much memory. For example, to make
> > > ~70k array elements, string_to_array seems to eat several
> Gig bytes
> > > of memory. ~70k array elements means there are same
> number of words
> > > in a document which is not too big in a large text IMO.
> >
> > Do you mean 70k unique lexemes? Ugh.
>
> I'm testing how GIN scales.
>
> > Why do not you use tsearch framework?
>
> ? I thought GIN is superior than tsearch2.
>
> From your GIN proposal posted to pgsql-hackers:
>
> "The primary goal of the Gin index is a scalable full text
> search in PostgreSQL"
tsearch2 *uses* GIN in 8.2. Just CREATE INDEX foo ON bar USING
gin(mytsvector).
And tsearch2 in 8.2 with GIN can be a *lot* faster than with GIST. I've
been running experiments on the website search with tsearch2/GIN and
i've been seeing fantastic performance compared top revious versions.
//Magnus
From | Date | Subject | |
---|---|---|---|
Next Message | Teodor Sigaev | 2006-11-08 10:03:40 | Re: string_to_array eats too much memory? |
Previous Message | Tatsuo Ishii | 2006-11-08 09:38:23 | Re: string_to_array eats too much memory? |