Re: Stop asserting that Unicode normalization consumes its whole input

On Mon Sep 21, 2026 at 11:46 PM CDT, John Naylor wrote:
> On Sat, Sep 19, 2026 at 12:00 AM Tristan Partin <tristan(at)partin(dot)io> wrote:
>>
>> normalize() and IS NORMALIZED determine how many code points their input
>> holds with pg_mbstrlen_with_len(). They then decode exactly that many
>> code points, and assert afterward that this consumed the entire datum.
>> That invariant does not hold because pg_mbstrlen_with_len() stops at the
>> first of "limit" or a NUL.
>
> The limit here is VARSIZE_ANY_EXHDR(input) , which is the entire
> length. This patch seems like it's weakening an assertion for a case
> that shouldn't happen. (I haven't looked at the tests in encoding.sql
> that force non-standard behavior, so I'm not sure what the context was
> there...)

Hey John,

In regress.c, we have the following function:

/* Convert bytea to text without validation for corruption tests from SQL. */
PG_FUNCTION_INFO_V1(test_bytea_to_text);
Datum
test_bytea_to_text(PG_FUNCTION_ARGS)
{
PG_RETURN_TEXT_P(PG_GETARG_BYTEA_PP(0));
}

In encoding.sql, we do things like the following:

CREATE TABLE regress_encoding(good text, truncated text, with_nul text, truncated_with_nul text);
INSERT INTO regress_encoding
VALUES ('café',
'caf' || test_bytea_to_text('\xc3'),
'café' || test_bytea_to_text('\x00') || 'dcba',
'caf' || test_bytea_to_text('\xc300') || 'dcba');
...
-- NUL = terminator
SELECT length(with_nul) FROM regress_encoding;
SELECT substring(with_nul, 3, 1) FROM regress_encoding;
SELECT substring(with_nul, 4, 1) FROM regress_encoding;
SELECT substring(with_nul, 5, 1) FROM regress_encoding;
SELECT convert_to(substring(with_nul, 5, 1), 'UTF8') FROM regress_encoding;
SELECT regexp_replace(with_nul, '^caf(.)$', '\1') FROM regress_encoding;
-- NUL = character
SELECT with_nul, reverse(with_nul), reverse(reverse(with_nul)) FROM regress_encoding;

Output file:

SELECT length(with_nul) FROM regress_encoding;
length
--------
4
(1 row)

SELECT substring(with_nul, 3, 1) FROM regress_encoding;
substring
-----------
f
(1 row)

SELECT substring(with_nul, 4, 1) FROM regress_encoding;
substring
-----------
é
(1 row)

SELECT substring(with_nul, 5, 1) FROM regress_encoding;
substring
-----------

(1 row)

SELECT convert_to(substring(with_nul, 5, 1), 'UTF8') FROM regress_encoding;
convert_to
------------
\x
(1 row)

SELECT regexp_replace(with_nul, '^caf(.)$', '\1') FROM regress_encoding;
regexp_replace
----------------
é
(1 row)

-- NUL = character
SELECT with_nul, reverse(with_nul), reverse(reverse(with_nul)) FROM regress_encoding;
with_nul | reverse | reverse
----------+---------+---------
café | abcd | café
(1 row)

You can see that we are doing some shenanigans to test how text values
work in various string functions for corrupted values. We don't do the
same tests for normalize() and IS NORMALIZED, which is why I made the
change to <= instead of ==. Otherwise, builds with casserts enabled will
crash, which doesn't seem like what we intend to do given the array of
tests we do to make sure things still work.

I would say that you're right that this patch does weaken an assertion,
but it is something that Postgres tests for just to make sure things
continue to work.

--
Tristan Partin
PostgreSQL Contributors Team
AWS (https://aws.amazon.com)