Skip site navigation (1) Skip section navigation (2)

Re: Can pg_trgm handle non-alphanumeric characters?

From: Fujii Masao <masao(dot)fujii(at)gmail(dot)com>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: MauMau <maumau307(at)gmail(dot)com>, Kevin Grittner <Kevin(dot)Grittner(at)wicourts(dot)gov>, Euler Taveira <euler(at)timbira(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: Can pg_trgm handle non-alphanumeric characters?
Date: 2012-05-11 15:53:57
Message-ID: (view raw, whole thread or download thread mbox)
Lists: pgsql-hackers
On Fri, May 11, 2012 at 4:11 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Fujii Masao <masao(dot)fujii(at)gmail(dot)com> writes:
>> On Fri, May 11, 2012 at 12:07 AM, MauMau <maumau307(at)gmail(dot)com> wrote:
>>> Thanks for your explanation. Although I haven't understood it well yet, I'll
>>> consider what you taught. And I'll consider if the tentative measure of
>>> removing KEEPONLYALNUM is correct for someone who wants to use pg_trgm
>>> against Japanese text.
>> In Japanese, it's common to do a text search with two characters keyword.
>> But since pg_trgm is 3-gram, you basically would not be able to use index
>> for such text search. So you might need something like pg_bigm or pg_unigm
>> for Japanese text search.

Even if an index can be used for two characters text search, bitmap index scan
picks up all rows, so it's too slow.

> I believe the trigrams are three *bytes* not three characters.  So a
> couple of kanji should work just fine for this.

Really? As far as I read the code of pg_trgm, the trigram is three characters
and its CRC32 is used as an index key if its size is more than three bytes.


Fujii Masao

In response to

pgsql-hackers by date

Next:From: Florian PflugDate: 2012-05-11 15:55:24
Subject: Re: Gsoc2012 idea, tablesample
Previous:From: Kevin GrittnerDate: 2012-05-11 15:50:37
Subject: Re: Gsoc2012 idea, tablesample

Privacy Policy | About PostgreSQL
Copyright © 1996-2017 The PostgreSQL Global Development Group