Re: Stop asserting that Unicode normalization consumes its whole input

On Sat, Sep 19, 2026 at 12:00 AM Tristan Partin <tristan(at)partin(dot)io> wrote:
>
> normalize() and IS NORMALIZED determine how many code points their input
> holds with pg_mbstrlen_with_len(). They then decode exactly that many
> code points, and assert afterward that this consumed the entire datum.
> That invariant does not hold because pg_mbstrlen_with_len() stops at the
> first of "limit" or a NUL.

The limit here is VARSIZE_ANY_EXHDR(input) , which is the entire
length. This patch seems like it's weakening an assertion for a case
that shouldn't happen. (I haven't looked at the tests in encoding.sql
that force non-standard behavior, so I'm not sure what the context was
there...)

--
John Naylor
Amazon Web Services