Re: Add ASCII fast path to Unicode normalization functions

On 2026-10-06 Tu 5:56 PM, Tristan Partin wrote:
> On Tue Oct 6, 2026 at 9:22 PM UTC, Andrew Dunstan wrote:
>> On 2026-10-04 Su 12:42 PM, Rui Zhao wrote:
>>> Hi Andrew,
>>>
>>> Thanks for working on this. I tried two further optimizations on top
>>> of v4. They can be applied independently:
>>>
>>> 1. unicode_assigned() still calls pg_mbstrlen_with_len() on the
>>> remaining input, then walks it again to check each code point. 0001
>>> uses the byte length to walk to the end directly. This also avoids
>>> scanning the whole suffix before returning false for an unassigned
>>> code point near the beginning.
>>>
>>> 2. unicode_normalize_func() and unicode_is_normalized() allocate a C
>>> string for the normalization form on every call, even when the input
>>> is all ASCII. 0002 compares the form directly in the text value,
>>> preserving case-insensitive matching and rejecting extra bytes.
>>> It only builds a C string when reporting an invalid form.
>>>
>>> 0003 replaces the constant-folded VALUES test with stored short and
>>> compressed text values.
>>>
>>> Core regression and the Unicode normalization checks passed with these
>>> patches.
>>>
>> OK, that all looks good. The attached combines all this.
> Thanks for the updates Andrew. They looks pretty good to me. A few
> nitpicks:
>
>> +/*
>> + * Return the length of an initial portion of s[0..len) that is valid ASCII,
>> + * that is, contains no zero bytes and no bytes with the high bit set, or
>> + * len if all of it is.
>> + *
>> + * is_valid_ascii() requires a length that is a multiple of sizeof(Vector8),
>> + * so check one chunk at a time with it and then the remainder byte by byte.
>> + * A failure inside a chunk is reported at the start of that chunk, so the
>> + * result can be up to sizeof(Vector8) - 1 bytes short of the full valid
>> + * ASCII prefix.
>> + */
> I would relegate that second paragraph to a comment within the function
> itself. I am not sure it is valuable to callers, only to those reading
> the actual implementation. It also may be a bit verbose, but that isjust
> an opinion.
>
>> + /*
>> + * Check the remaining code points without first counting them, stopping
>> + * at a NUL as pg_mbstrlen_with_len() would.
>> + */
>> + p = (unsigned char *) VARDATA_ANY(input) + start;
>> + end = (unsigned char *) VARDATA_ANY(input) + len;
> I am not sure that it makes sense to mention pg_mbstrlen_with_len()
> here. I would be more explicit as to why we are stopping at a NUL.

Thanks. I'm intending to commit this during the November CF, will fix then.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com