BUG #19860: LDAP authentication fails for non-ASCII passwords and usernames on Windows (UTF-8 bytes passed to th

The following bug has been logged on the website:

Bug reference: 19860
Logged by: Manuela Texier
Email address: manuela(dot)texier(at)veremes(dot)com
PostgreSQL version: 18.6
Operating system: Windows 11 24H2, French locale, system ANSI code p
Description:

Summary
-------
On Windows, LDAP authentication fails for any password or username
containing
non-ASCII characters. The bytes received from the client are passed
unchanged to
ldap_simple_bind_s(). On Windows that symbol is the neutral form of the
wldap32
entry point, which converts its arguments from the system ANSI code page to
UTF-16 before encoding them as UTF-8 for the wire. Since the client sent
UTF-8,
the result is doubly encoded: the directory receives a different password
than
the user typed, and the bind fails with "Invalid Credentials".

Tested on 18.6 and on 18.3, with identical results byte for byte.

Steps to reproduce
------------------
1. Start a minimal LDAP listener that logs the bytes of the simple bind
request
and always answers success (~60 lines of Python; it only needs to parse
the
BindRequest and return a BindResponse with resultCode 0). Have it listen
on
127.0.0.1:3890. I can provide it on request.

2. Optionally, put a TCP relay in front of PostgreSQL to log the
PasswordMessage
sent by the client. This isolates libpq from the backend.

3. initdb -D data -E UTF8 --locale=C -U postgres -A trust

4. In data/pg_hba.conf, before any other rule:
host all testuser 127.0.0.1/32 ldap ldapserver=127.0.0.1 ldapport=3890
ldaptls=0 ldapprefix=""

5. Start the server, then: CREATE ROLE testuser LOGIN;

6. Connect as testuser with the password Tèst2026, passing the exact UTF-8
bytes
54 c3 a8 73 74 32 30 32 36 (9 bytes). I called PQconnectdb() directly so
that
the bytes are not altered by command-line or environment conversion.

Actual result
-------------
The relay shows the PasswordMessage carrying those 9 bytes unchanged, so
libpq
transmits them intact. The LDAP listener then receives 11 bytes:

54 c3 83 c2 a8 73 74 32 30 32 36

The two UTF-8 bytes c3 a8 of "è" were each read as a separate CP1252
character
("Ã" and "¨") and re-encoded as UTF-8, yielding c3 83 c2 a8.

Expected result
---------------
The listener should receive the bytes the client sent,
54 c3 a8 73 74 32 30 32 36, so that the directory sees the password the user
actually typed.

Controls
--------
- An ASCII-only password passes through both stages unchanged, so the
transformation only affects bytes >= 0x80.

- The code page is CP1252, not Latin-1: with the password Prix20€, the bytes
e2 82 ac arrive as c3 a2 e2 80 9a c2 ac. The 0x82 byte becomes U+201A,
which
exists only in CP1252; Latin-1 would have produced c2 82.

- The role name is affected identically. "tèstuser" sent as
74 c3 a8 73 74 75 73 65 72 arrives as 74 c3 83 c2 a8 73 74 75 73 65 72,
since fulluser and passwd are the two arguments of the same call.

- Supplying the CP1252 form instead (54 e8 73 74 32 30 32 36) makes the
listener
receive the correct UTF-8 form, confirming a single ANSI-to-UTF-8
conversion.

- On the same cluster, the same accented password authenticates successfully
with scram-sha-256, so the bytes reach the backend intact.

Note: the listener always answers success, so I measured the byte
corruption,
not the authentication failure itself. The failure against a real directory
is
inferred from it.

Analysis
--------
src/backend/libpq/auth.c calls:

r = ldap_simple_bind_s(ldap, fulluser, passwd);

passwd comes from recv_password_packet(), which documents that it performs
no
conversion ("we do not attempt to do any character-set conversion on it;
since
we don't yet know the client's encoding, there wouldn't be much point").

On Windows the file includes <winldap.h> and UNICODE is not defined, so
LDAP_UNICODE is 0 and ldap_simple_bind_s is declared taking PSTR arguments.
The
header's own comment states that this form "simply converts the arguments to
unicode and calls ldap_bindW", because "we have to put UTF-8 on the wire".
That
conversion uses the system ANSI code page, which is what corrupts UTF-8
input.

wldap32.dll exports three distinct forms at distinct ordinals:
ldap_simple_bind_s ordinal 60
ldap_simple_bind_sA ordinal 215
ldap_simple_bind_sW ordinal 216

The shipped postgres.exe imports ordinal 60, the neutral converting form.
Versions 13, 14, 16, 18.3 and 18.6 all import that same ordinal.

The call site is unchanged in REL_16_STABLE, REL_17_STABLE, REL_18_STABLE,
REL_19_STABLE and master (20devel), so I expect all supported versions to
behave
the same, although I only executed 18.3 and 18.6.

I could not find an existing report describing this mechanism.

Possible fix
------------
On Windows, pass the credentials through ldap_sasl_bind_s() with a berval
structure, which winldap.h documents as not being translated ("so that it
isn't
translated from Unicode or ANSI to UTF8"). Unlike switching to the W
variants,
this does not require the server to know the client's encoding. The same
applies
to fulluser, and to the bind DN in search+bind mode.

This would also explain an earlier report that was never diagnosed:
BUG #8456 / #8457 (2013-09-16) - LDAP authentication fails for an accented
login, PostgreSQL 9.0.6 on Windows 8 with Active Directory