Add ASCII fast path to Unicode normalization functions

From: Andrew Dunstan <andrew(at)dunslane(dot)net>
To: PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org>
Subject: Add ASCII fast path to Unicode normalization functions
Date: 2026-09-14 14:50:45
Message-ID: 8cb366d6-a30f-49a4-b5d3-a5dfc0f2fea2@dunslane.net
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hi,

A linkedin post comparing CedarDB's new Unicode normalization support
to PostgreSQL's caught my eye [1]: same results, but a claimed 30x
speedup on "SELECT count(*) FROM hits WHERE url IS NORMALIZED" over
ClickBench's hits table. Most of that turned out to be down to CedarDB
using all available threads by default versus our
max_parallel_workers_per_gather of 2. But even at the matched thread
count they reported a 6x edge, attributed to two things: an ASCII fast
path (most URLs are already normalized ASCII, so you can skip decoding
entirely), and vectorized byte scanning for the ASCII check itself.

I went and looked, and unicode_is_normalized(), unicode_assigned(), and
normalize() all decode every string to an array of char32_t codepoints,
one utf8_to_unicode()/pg_utf_mblen() call at a time, before doing any
real work -- including on input that's already pure ASCII. The attached
patch adds a fast path: scan the raw bytes for anything with the high
bit set, using the SIMD-vectorized is_valid_ascii() we already have
(currently only used inside pg_utf8_verifystr()). If nothing is found,
the string is trivially normalized (ASCII code points have no
canonical or compatibility decomposition, and a combining class of
zero) and every code point in it is assigned, so all three functions
can return immediately.

I deliberately didn't copy CedarDB's trick of comparing byte length to
codepoint count -- getting the codepoint count means calling
pg_mbstrlen_with_len(), exactly the scalar work this patch avoids.
Scanning raw bytes with is_valid_ascii() instead reuses SIMD
infrastructure we already have, and is cheaper to begin with: a single
reduction versus a population count.

Benchmarked with data sized to fit comfortably under shared_buffers rather
than triggering the seqscan ring-buffer bypass, which otherwise swamps the
comparison at larger table sizes: ~10x on pure ASCII, ~4x on an 85/15
ASCII/non-ASCII mix, and no measurable regression on non-ASCII input
that still needs the full decode-and-quickcheck path.

Regression tests cover the ASCII-hit case for all three functions, plus
a boundary sweep that plants a non-NFC sequence at varying offsets
around ASCII padding, to catch any off-by-one in the SIMD-chunk/scalar-
remainder split.

cheers

andrew

[1] https://lnkd.in/p/eKUqSj73

--
Andrew Dunstan
EDB: https://www.enterprisedb.com

Attachment Content-Type Size
0001-Add-ASCII-fast-path-to-Unicode-normalization-functio.patch text/x-patch 9.7 KB

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Andres Freund 2026-09-14 14:56:12 Re: ExecForceStoreHeapTuple() loses tts_tid, so ORDER BY-op index scans project an invalid ctid
Previous Message Rui Zhao 2026-09-14 14:35:55 Re: index prefetching