Re: snowball ASCII stemmer configuration

> On Jun 16, 2020, at 7:37 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>
> I wrote:
>> Peter Eisentraut <peter(dot)eisentraut(at)2ndquadrant(dot)com> writes:
>>> Moreover, AFAIK, the following other languages do not use Latin-based
>>> alphabets:
>
>>> arabic arabic \
>>> greek greek \
>>> nepali nepali \
>>> tamil tamil \
>
>> Hmm. I think all of those entries are ones that got added by me while
>> absorbing post-2007 Snowball updates, and I confess that I did not think
>> about this point. Maybe these should be changed.
>
> After further reflection, I think these are indeed mistakes and we should
> change them all. The argument for the Russian/English case, AIUI, is
> "if we come across an all-ASCII word, it is most certainly not Russian,
> and the most likely Latin-based language is English". Given the world
> as it is, I think the same argument works for all non-Latin-alphabet
> languages. Obviously specific applications might have a different idea
> of the best fallback language, but that's why we let users make their
> own text search configurations. For general-purpose use, falling back
> to English seems reasonable. And we can be dead certain that applying
> a Greek stemmer to an ASCII word will do nothing useful, so the
> configuration choice shown above is unhelpful.

I am a bit surprised to see that you are right about this, because non-latin languages often have transliteration/romanization schemes for writing the language in the Latin alphabet, developed before computers had wide spread adoption of non-ASCII character sets, and still in use today for text messaging. I expected to find stemming rules for transliterated words, but can't find any indication of that, neither in the postgres sources, nor in the snowball sources I pulled from their repo. Is there some architectural separation of stemming from transliteration such that we'd never need to worry about it? If snowball ever published stemmers for transliterated text, we might have to revisit this issue, but for now your proposed change sounds fine to me.

—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company