text search: restricting the number of parsed words in headline generation

From: Sushant Sinha <sushant354(at)gmail(dot)com>
To: pgsql-hackers(at)postgresql(dot)org
Subject: text search: restricting the number of parsed words in headline generation
Date: 2011-08-23 16:40:20
Message-ID: 1314117620.3700.12.camel@dragflick
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Given a document and a query, the goal of headline generation is to
produce text excerpts in which the query appears. Currently the headline
generation in postgres follows the following steps:

1. Tokenize the documents and obtain the lexemes
2. Decide on lexemes that should be the part of the headline
3. Generate the headline

So the time taken by the headline generation is directly dependent on
the size of the document. The longer the document, the more time taken
to tokenize and more lexemes to operate on.

Most of the time is taken during the tokenization phase and for very big
documents, the headline generation is very expensive.

Here is a simple patch that limits the number of words during the
tokenization phase and puts an upper-bound on the headline generation.
The headline function takes a parameter MaxParsedWords. If this
parameter is negative or not supplied, then the entire document is
tokenized and operated on (the current behavior). However, if the
supplied MaxParsedWords is a positive number, then the tokenization
stops after MaxParsedWords is obtained. The remaining headline
generation happens on the tokens obtained till that point.

The current patch can be applied to 9.1rc1. It lacks changes to the
documentation and test cases. I will add them if you folks agree on the
functionality.

-Sushant.

Attachment Content-Type Size
9.1rc1_max_parsed_words.patch text/x-patch 2.6 KB

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Dave Cramer 2011-08-23 16:48:01 Why doesn't psql use the information schema to get ACL description ?
Previous Message Dimitri Fontaine 2011-08-23 16:39:30 Re: cheaper snapshots redux