Re: Windows UTF-8, non-ICU collation trouble

On Fri, Dec 06, 2019 at 07:56:08PM +1300, Thomas Munro wrote:
> On Fri, Dec 6, 2019 at 7:34 PM Noah Misch <noah(at)leadboat(dot)com> wrote:
> > We use system UTF-16 collation to implement UTF-8 collation on Windows. The
> > PostgreSQL security team received a report, from Timothy Kuun, that this
> > collation does not uphold the "symmetric law" and "transitive law" that we
> > require for btree operator classes. The attached test program demonstrates
> > this. http://www.delphigroups.info/2/62/478610.html quotes reports of that
> > problem going back eighteen years. Most code points are unaffected. Indexing
> > an affected code point using such a collation can cause btree index scans to not
> > find a row they should find and can make a UNIQUE or PRIMARY KEY constraint
> > admit a duplicate. The security team determined that this doesn't qualify as a
> > security vulnerability, but it's still a bug.
>
> Huh. Does this apply in modern times? Since Windows 10, I thought
> they adopted[1] CLDR data to drive that, the same definitions used (or
> somewhere in the process of being adopted by) GNU, Illumos, FreeBSD
> etc. Basically, everyone gave up on trying to own this rats nest of a
> problem and deferred to the experts.

Based on my test program, it applies to Windows Server 2016. I didn't test
newer versions.

> If you can still get
> index-busting behaviour out of modern Windows collations, wouldn't
> that be a bug that someone can file against SQL Server, Windows etc
> and get fixed?

Perhaps. I wouldn't have high hopes, given the behavior's long tenure and the
risk of breaking a different set of applications.