Re: BUG #15548: Unaccent does not remove combining diacritical characters

From: Ramanarayana <raam(dot)soft(at)gmail(dot)com>
To: Hugh Ranalli <hugh(at)whtc(dot)ca>
Cc: pgsql-hackers(at)lists(dot)postgresql(dot)org
Subject: Re: BUG #15548: Unaccent does not remove combining diacritical characters
Date: 2019-02-11 20:57:31
Views: Raw Message | Whole Thread | Download mbox
Lists: pgsql-bugs pgsql-hackers

Hi Hugh,

I tested the script in python 2.7 and it works perfect. The problem is in
python 3.7(and may be only in windows as you were not getting the issue)
and I was getting the following error

UnicodeEncodeError: 'charmap' codec can't encode character '\u0100' in
position 0: character maps to <undefined>

I went through the python script and found that the stdout encoding is set
to utf-8 only if python version is <=2.

I have made the same change for python version 3 as well. Please find the
patch for the same.Let me know if it makes sense


On Tue, 12 Feb 2019 at 00:50, Hugh Ranalli <hugh(at)whtc(dot)ca> wrote:

> On Sun, 10 Feb 2019 at 15:07, raam narayana <raam(dot)soft(at)gmail(dot)com> wrote:
>> Hi,
>> After the latest commit in master branch, I was trying to test the python
>> script. Ironically I still see that the output from the script is
>> completely different from the unaccent.rules file content. Am I missing
>> anything.My testing includes the following
>> Downloaded the following files
>> Executed the below python script
>> python --unicode-data-file UnicodeData.txt
>> --latin-ascii-file Latin-ASCII.xml > unaccent.rules
>> I am using python 3.7.1 and running on Windows 10 Platform
>> The new status of this patch is: Needs review
> Hi Raam,
> I just ran under two environments, using the
> data files given above :
> - Python 3.4.3 on Linux Mint 17.3 (equivalent to Ubuntu 14.04)
> - Python 3.6.7 on Ubuntu 18.04
> In both cases, the output was identical to that generated by the program
> under Python 2.7. So yes, more information would help. Unfortunately I
> don't have a Windows Python environment readily available, but could set
> one up if I had to.
> Thanks,
> Hugh

Ram 4.0

Attachment Content-Type Size
generate_unaccent_rules-remove-combining-diacritical-accents-03.patch application/octet-stream 544 bytes

In response to


Browse pgsql-bugs by date

  From Date Subject
Next Message Saul, Jean Paolo 2019-02-12 00:27:49 Re: BUG #15609: synchronous_commit=off insert performance regression with secondary indexes
Previous Message Tom Lane 2019-02-11 20:33:08 Re: BUG #15631: Generated as identity field in a temporary table with on commit drop corrupts system catalogs

Browse pgsql-hackers by date

  From Date Subject
Next Message Julien Rouhaud 2019-02-11 20:59:59 Re: Inadequate executor locking of indexes
Previous Message Peter Geoghegan 2019-02-11 20:54:30 Re: Making all nbtree entries unique by having heap TIDs participate in comparisons