| From: | Andrew Dunstan <andrew(at)dunslane(dot)net> |
|---|---|
| To: | PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org> |
| Subject: | Add ASCII fast path to Unicode normalization functions |
| Date: | 2026-09-14 14:50:45 |
| Message-ID: | 8cb366d6-a30f-49a4-b5d3-a5dfc0f2fea2@dunslane.net |
| Views: | Whole Thread | Raw Message | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-hackers |
Hi,
A linkedin post comparing CedarDB's new Unicode normalization support
to PostgreSQL's caught my eye [1]: same results, but a claimed 30x
speedup on "SELECT count(*) FROM hits WHERE url IS NORMALIZED" over
ClickBench's hits table. Most of that turned out to be down to CedarDB
using all available threads by default versus our
max_parallel_workers_per_gather of 2. But even at the matched thread
count they reported a 6x edge, attributed to two things: an ASCII fast
path (most URLs are already normalized ASCII, so you can skip decoding
entirely), and vectorized byte scanning for the ASCII check itself.
I went and looked, and unicode_is_normalized(), unicode_assigned(), and
normalize() all decode every string to an array of char32_t codepoints,
one utf8_to_unicode()/pg_utf_mblen() call at a time, before doing any
real work -- including on input that's already pure ASCII. The attached
patch adds a fast path: scan the raw bytes for anything with the high
bit set, using the SIMD-vectorized is_valid_ascii() we already have
(currently only used inside pg_utf8_verifystr()). If nothing is found,
the string is trivially normalized (ASCII code points have no
canonical or compatibility decomposition, and a combining class of
zero) and every code point in it is assigned, so all three functions
can return immediately.
I deliberately didn't copy CedarDB's trick of comparing byte length to
codepoint count -- getting the codepoint count means calling
pg_mbstrlen_with_len(), exactly the scalar work this patch avoids.
Scanning raw bytes with is_valid_ascii() instead reuses SIMD
infrastructure we already have, and is cheaper to begin with: a single
reduction versus a population count.
Benchmarked with data sized to fit comfortably under shared_buffers rather
than triggering the seqscan ring-buffer bypass, which otherwise swamps the
comparison at larger table sizes: ~10x on pure ASCII, ~4x on an 85/15
ASCII/non-ASCII mix, and no measurable regression on non-ASCII input
that still needs the full decode-and-quickcheck path.
Regression tests cover the ASCII-hit case for all three functions, plus
a boundary sweep that plants a non-NFC sequence at varying offsets
around ASCII padding, to catch any off-by-one in the SIMD-chunk/scalar-
remainder split.
cheers
andrew
[1] https://lnkd.in/p/eKUqSj73
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
| Attachment | Content-Type | Size |
|---|---|---|
| 0001-Add-ASCII-fast-path-to-Unicode-normalization-functio.patch | text/x-patch | 9.7 KB |
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Andres Freund | 2026-09-14 14:56:12 | Re: ExecForceStoreHeapTuple() loses tts_tid, so ORDER BY-op index scans project an invalid ctid |
| Previous Message | Rui Zhao | 2026-09-14 14:35:55 | Re: index prefetching |