Re: [PATCH] Completed unaccent dictionary with many missing characters

From: Przemysław Sztoch <przemyslaw(at)sztoch(dot)pl>
To: Michael Paquier <michael(at)paquier(dot)xyz>
Cc: pgsql-hackers(at)lists(dot)postgresql(dot)org, Peter Eisentraut <peter(dot)eisentraut(at)enterprisedb(dot)com>
Subject: Re: [PATCH] Completed unaccent dictionary with many missing characters
Date: 2022-06-21 13:36:49
Message-ID: 557e590f-d3af-e516-a22d-c4dd9736317c@sztoch.pl
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Michael Paquier wrote on 21.06.2022 02:11:
> On Mon, Jun 20, 2022 at 10:37:57AM +0200, Przemysław Sztoch wrote:
>> But ligature check is performed on combining_ids (result of translation),
>> not on base codepoint.
>> Without it, you will get assertions in get_plain_letters.
> Hmm. I am wondering if we could make the whole logic a bit more
> intuitive here. The loop that builds the set of mappings gets now
> much more complicated with the addition of the categories beginning by
> N for the numbers, and that's mostly the same set of checks as the
> ones applied for T.
I'm sorry, but I can't correct this condition.
I have tried, but there are further exceptions and errors.
>
>> However, the current Latin-ASCII.xml suggests a conversion to x.
>> I found an open discussion on the internet about this and the suggestion
>> that the Latin-ASCII.xml file should be corrected for this letter.
>> But I wouldn't expect that Unicode makes the revised Latin-ASCII.xml quickly
>> into the official repo.
> Yeah, Latin-ASCII.xml is getting it wrong here, then. unaccent
> fetches the thing from this URL currently:
> https://raw.githubusercontent.com/unicode-org/cldr/release-41/common/transforms/Latin-ASCII.xml
>
> Could it be better to handle that as an exception in
> generate_unaccent_rules.py, documenting why we are doing it this way
> then? My concern is somebody re-running the script without noticing
> this exception, and the set of rules would be blindly, and
> incorrectly, updated.
I replaced python set with python dictionary.
It resolve problem with duplicated entry.
I left the conversion to "x". It was like that before and I leave it as
it was.
The conversion to "x" is probably due to the phonetic interpretation of
this sign.
If they correct the Latin-ASCII.xml file, it will change.
> --
> Michael

--
Przemysław Sztoch | Mobile +48 509 99 00 66

Attachment Content-Type Size
unaccent-missed-chars-v3.patch text/plain 19.4 KB

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Przemysław Sztoch 2022-06-21 13:41:48 Re: [PATCH] Completed unaccent dictionary with many missing characters
Previous Message Aleksander Alekseev 2022-06-21 13:33:03 Re: doc: array_length produces null instead of 0