Stop asserting that Unicode normalization consumes its whole input

From: "Tristan Partin" <tristan(at)partin(dot)io>
To: "pgsql-hackers" <pgsql-hackers(at)lists(dot)postgresql(dot)org>
Subject: Stop asserting that Unicode normalization consumes its whole input
Date: 2026-09-18 17:00:32
Message-ID: DLILPXVKR2GW.1KP2DGOPCP7P2@partin.io
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

normalize() and IS NORMALIZED determine how many code points their input
holds with pg_mbstrlen_with_len(). They then decode exactly that many
code points, and assert afterward that this consumed the entire datum.
That invariant does not hold because pg_mbstrlen_with_len() stops at the
first of "limit" or a NUL.

Postgres doesn't allow text types to contain NUL, so reaching this
requires work, but src/test/regress/sql/encoding.sql already constructs
such values to pin down how the string functions behave on them. It just
never passed them to normalize() or IS NORMALIZED, which is why this
went unnoticed.

--
Tristan Partin
PostgreSQL Contributors Team
AWS (https://aws.amazon.com)

Attachment Content-Type Size
v1-0001-Stop-asserting-that-Unicode-normalization-consume.patch text/x-patch 3.4 KB

Browse pgsql-hackers by date

  From Date Subject
Next Message Hannu Krosing 2026-09-18 17:01:01 Re: ANSI SQL proposal: SELECT DISTINCT ON (... ORDER BY ...) and UNION DISTINCT ON (... ORDER BY ...)
Previous Message shihao zhong 2026-09-18 16:58:55 Re: [patch] Cache invalidation for I/O Workers