From: | Andrew Dunstan <andrew(at)dunslane(dot)net> |
---|---|
To: | Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> |
Cc: | Heikki Linnakangas <heikki(dot)linnakangas(at)enterprisedb(dot)com>, pgsql-hackers(at)postgreSQL(dot)org |
Subject: | Re: Notes about fixing regexes and UTF-8 (yet again) |
Date: | 2012-02-17 14:56:05 |
Message-ID: | 4F3E6A85.3090606@dunslane.net |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-hackers |
On 02/17/2012 09:39 AM, Tom Lane wrote:
> Heikki Linnakangas<heikki(dot)linnakangas(at)enterprisedb(dot)com> writes:
>> Here's a wild idea: keep the class of each codepoint in a hash table.
>> Initialize it with all codepoints up to 0xFFFF. After that, whenever a
>> string contains a character that's not in the hash table yet, query the
>> class of that character, and add it to the hash table. Then recompile
>> the whole regex and restart the matching engine.
>> Recompiling is expensive, but if you cache the results for the session,
>> it would probably be acceptable.
> Dunno ... recompiling is so expensive that I can't see this being a win;
> not to mention that it would require fundamental surgery on the regex
> code.
>
> In the Tcl implementation, no codepoints above U+FFFF have any locale
> properties (alpha/digit/punct/etc), period. Personally I'd not have a
> problem imposing the same limitation, so that dealing with stuff above
> that range isn't really a consideration anyway.
up to U+FFFF is the BMP which is described as containing "characters for
almost all modern languages, and a large number of special characters."
It seems very likely to be acceptable not to bother about the locale of
code points in the supplementary planes.
See <http://en.wikipedia.org/wiki/Plane_%28Unicode%29> for descriptions
of which sets of characters are involved.
cheers
andrew
From | Date | Subject | |
---|---|---|---|
Next Message | Robert Haas | 2012-02-17 15:12:22 | Re: Notes about fixing regexes and UTF-8 (yet again) |
Previous Message | Tom Lane | 2012-02-17 14:39:43 | Re: Notes about fixing regexes and UTF-8 (yet again) |