Re: [Fwd: Re: tsearch in core patch]

> Ishii-san,
>
> >>> Ok, probably we need to copy the English stemming rule to the one for
> >>> Japanese.
> >> Pardon my ignorance here, but is the concept of stemming even relevant
> >> to Japanese/Chinese/Korean? What little I know about ideographic
> >> languages suggests it wouldn't work well. And surely the specific rules
> >> in the Snowball project's English stemmer wouldn't work.
> >
> > Your undestanding is correct. English stemmer would not work for
> > Japanese "non English" part.
>
> That reminds me, don't you guys have your own full text search for
> Japanese? Planning on merging it with the core code anytime soon?

No. Actually Japanese (non English part) does not need stemming at
all. However, since Japanese is an agglutinative language, we have to
break continuous Japanese string into space separated "words". For
example, we need to break:

todayisfine

into:

today is fine

(of course those English are just for non-Japanese spearker's
understanding, actually they are Japanese).

For this we need good dictionary and software. Fortunately we have
several kinds of open source softwares for this pupose. Once I have
written a PostgreSQL C function envoking one of these software to do
the work and it works great with tsearch2.
--
Tatsuo Ishii
SRA OSS, Inc. Japan