Re: Radix tree for character conversion

On 10/07/2016 06:55 PM, Robert Haas wrote:
> On Fri, Oct 7, 2016 at 6:46 AM, Heikki Linnakangas <hlinnaka(at)iki(dot)fi> wrote:
>> Ouch. We should find and document an authoritative source for all the
>> mappings we have...
>>
>> I think the next steps here are:
>>
>> 1. Find an authoritative source for all the existing mappings.
>> 2. Generate the radix tree files directly from the authoritative sources,
>> instead of the existing *.map files.
>> 3. Completely replace the existing binary-search code with this.
>
> It might be best to convert using the existing map files, and then
> update the mappings later. Otherwise, when things break, you won't
> know what to blame.

I was thinking that we keep the mappings unchanged, but figure out where
we got the mappings we have. An authoritative source may well be "file X
from unicode, with the following tweaks: ...". As long as we have some
way of representing that, in text files, or in perl code, that's OK.

What I don't want is that the current *.map files are turned into the
authoritative source files, that we modify by hand. There are no
comments in them, for starters, which makes hand-editing cumbersome. It
seems that we have edited some of them by hand already, but we should
rectify that.

- Heikki