Re: [PATCH] Fix SASLprep validation after NFKC normalization

Hi,

Attached is v2 of the patch, along with a Python script I wrote to test PostgreSQL's SASLprep implementation against
Python's Unicode 3.2 implementation.

The updated patch addresses two issues:

1. Prohibited output characters and bidirectional properties must be checked after NFKC normalization, using the
normalized string rather than the original input.

2. Code points unassigned in Unicode 3.2 must be rejected before normalization. PostgreSQL's normalizer uses newer
Unicode data, which can normalize some previously unassigned code points into characters that were already assigned
in Unicode 3.2. Checking only after normalization would allow these characters to bypass the unassigned code point
check.

I've also added 13 regression test cases covering both issues. Nine of these fail with the original implementation,
while all 13 pass with the patch.

Additional testing

To verify the changes more thoroughly, I wrote a Python script that uses unicodedata.ucd_3_2_0 and stringprep as a
reference implementation of SASLprep.

The script tests all Unicode scalar values (excluding U+0000 and surrogates), along with selected combinations intended
to exercise normalization and bidirectional rules.

It checks 1,112,210 test cases in total.

The results were:

- Original PostgreSQL implementation: 104 mismatches

- Initial patch: 704 mismatches

- Updated patch (v2): 5 mismatches

The increase with the initial patch revealed the problem with checking unassigned code points only after normalization,
which is now addressed in v2.

The five remaining mismatches are all normalization differences involving the code points U+2F868, U+2F874, U+2F91F,
U+2F95F, and U+2F9BF. These correspond to the five mappings corrected by Unicode Corrigendum #4:

https://www.unicode.org/versions/corrigendum4.html

In all five cases, both PostgreSQL and the reference implementation accept the input, but produce different normalized output.

Test script

I've attached saslprep_diff.py as a standalone testing tool. It requires Python 3 and psycopg, and runs against a
PostgreSQL instance with the test_saslprep extension installed.

I'm not sure how best to integrate this kind of exhaustive testing into PostgreSQL's existing test infrastructure,
especially since the script currently depends on Python and psycopg.

For now, I've included the representative test cases in the regular SQL regression tests and attached the Python script
separately so the more extensive testing can be reproduced.

I'd be happy to adapt the script or its generated test cases if there is a preferred way to include this kind of
coverage in the existing test suite.

I understand that the focus is currently on getting v19 ready, so there is no urgency for a review. I just wanted to make the updated patch and test results available.

Thanks!