multi terabyte fulltext searching

From: Benjamin Arai <me(at)benjaminarai(dot)com>
To: Postgresql <pgsql-general(at)postgresql(dot)org>
Subject: multi terabyte fulltext searching
Date: 2007-03-21 14:43:43
Message-ID: 5C6E1151-F51C-4486-BB64-5F2CDE08ADB7@benjaminarai.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

Hi,

I have been struggling with getting fulltext searching for very large
databases. I can fulltext index 10s if gigs without any problem but
when I start geting to hundreds of gigs it becomes slow. My current
system is a quad core with 8GB of memory. I have the resource to
throw more hardware at it but realistically it is not cost effective
to buy a system with 128GB of memory. Is there any solutions that
people have come up with for indexing very large text databases?

Essentially I have several terabytes of text that I need to index.
Each record is about 5 paragraphs of text. I am currently using
TSearch2 (stemming and etc) and getting sub-optimal results. Queries
take more than a second to execute. Has anybody implemented such a
database using multiple systems or some special add-on to TSearch2 to
make things faster? I want to do something like partitioning the
data into multiple systems and merging the ranked results at some
master node. Is something like this possible for PostgreSQL or must
it be a software solution?

Benjamin

Browse pgsql-general by date

  From Date Subject
Next Message Joshua D. Drake 2007-03-21 14:46:00 Re: Lifecycle of PostgreSQL releases
Previous Message Leticia 2007-03-21 14:29:54 CHAR data type