Re: Add ASCII fast path to Unicode normalization functions

> On Sep 14, 2026, at 22:50, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
>
> Hi,
>
> A linkedin post comparing CedarDB's new Unicode normalization support
> to PostgreSQL's caught my eye [1]: same results, but a claimed 30x
> speedup on "SELECT count(*) FROM hits WHERE url IS NORMALIZED" over
> ClickBench's hits table. Most of that turned out to be down to CedarDB
> using all available threads by default versus our
> max_parallel_workers_per_gather of 2. But even at the matched thread
> count they reported a 6x edge, attributed to two things: an ASCII fast
> path (most URLs are already normalized ASCII, so you can skip decoding
> entirely), and vectorized byte scanning for the ASCII check itself.
>
> I went and looked, and unicode_is_normalized(), unicode_assigned(), and
> normalize() all decode every string to an array of char32_t codepoints,
> one utf8_to_unicode()/pg_utf_mblen() call at a time, before doing any
> real work -- including on input that's already pure ASCII. The attached
> patch adds a fast path: scan the raw bytes for anything with the high
> bit set, using the SIMD-vectorized is_valid_ascii() we already have
> (currently only used inside pg_utf8_verifystr()). If nothing is found,
> the string is trivially normalized (ASCII code points have no
> canonical or compatibility decomposition, and a combining class of
> zero) and every code point in it is assigned, so all three functions
> can return immediately.
>
> I deliberately didn't copy CedarDB's trick of comparing byte length to
> codepoint count -- getting the codepoint count means calling
> pg_mbstrlen_with_len(), exactly the scalar work this patch avoids.
> Scanning raw bytes with is_valid_ascii() instead reuses SIMD
> infrastructure we already have, and is cheaper to begin with: a single
> reduction versus a population count.
>
>
> Benchmarked with data sized to fit comfortably under shared_buffers rather
> than triggering the seqscan ring-buffer bypass, which otherwise swamps the
> comparison at larger table sizes: ~10x on pure ASCII, ~4x on an 85/15
> ASCII/non-ASCII mix, and no measurable regression on non-ASCII input
> that still needs the full decode-and-quickcheck path.
>
> Regression tests cover the ASCII-hit case for all three functions, plus
> a boundary sweep that plants a non-NFC sequence at varying offsets
> around ASCII padding, to catch any off-by-one in the SIMD-chunk/scalar-
> remainder split.
>
>
> cheers
>
>
> andrew
>
>
> [1] https://lnkd.in/p/eKUqSj73
>
> --
> Andrew Dunstan
> EDB: https://www.enterprisedb.com
> <0001-Add-ASCII-fast-path-to-Unicode-normalization-functio.patch>

The patch looks good to me.

I also did some benchmark testing on my MacBook Air M4. I used clean builds with -O2 and without -g.

# unicode_is_normalized()

* all ascii: master 572ms; patch 68ms; Huge improvement
* mixed: master 529ms; patch 64ms; Big improvement
* non-ascii: master 399ms; patch 397ms; Roughly unchanged
* late-non-ascii: master 1069ms; patch 1074ms; Roughly unchanged; This is the worse case, most of chars are ascii, and only unicode appear in the end

# unicode_normalize_func()

* all ascii: master 1659ms; patch 68ms; Huge improvement
* mixed: master 1909ms; patch 233ms; Big improvement
* non-ascii: master 1432ms; patch 1452ms; Roughly unchanged
* late-non-ascii: master 3572ms; patch 3578ms; Roughly unchanged

# unicode_assigned()

* all ascii: master 258ms; patch 60ms; Big improvement
* mixed: master 237ms; patch 61ms; Big improvement
* non-ascii: master 221ms; patch 219ms; Roughly unchanged
* late-non-ascii: master 531ms; patch 533ms; Roughly unchanged

The late-non-ascii case is intended to be a worst case for the added ascii scan: most of the string is ascii, with the first non-ascii character appearing near the end.

For pure ascii and mixed ascii/non-ascii input, all three functions show substantial improvements. For pure non-ascii input, including the late-non-ascii case, performance is roughly unchanged.

So this looks like a worthwhile performance optimization to me.

The attached is my test script.

Best regards,
--
Chao Li (Evan)
HighGo Software Co., Ltd.
https://www.highgo.com/