Re: BUG #1931: ILIKE and LIKE fails on Turkish locale

Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:

> Victor Snezhko <snezhko(at)indorsoft(dot)ru> writes:
>> 2) When I try to create a stored procedure to create table (e.g., for
>> creating table only if it doesn't already exist), it fails to
>> compile if i use cyrillic letter "х" (unicode id: 0x0445, utf-8
>> representation is D1 85) with the following weird error:
>
>> ERROR: expected dot between identifiers: х
>> CONTEXT: compile of PL/pgSQL function "createoraltertable" near line 2
>
>> the following query fails:
>
>> CREATE OR REPLACE FUNCTION TestFunction()
>> RETURNS int
>> AS $$
>> BEGIN
>> SELECT х FROM test_table;
>> RETURN 0;
>> END;
>> $$ LANGUAGE plpgsql;
>
> I tried to duplicate this without success on Fedora Core 5. Either your
> ru_RU.utf8 locale works differently from Fedora's, or something else is
> wrong.
> Looking at the code, the only very plausible theory is that
> isspace() is doing something different than we expect it to. I wonder
> whether you have the database encoding set to something else than what
> the postmaster's LC_CTYPE locale expects?

No, the database encoding is UTF-8, the cluster was built with
ru_RU.UTF-8 locale, and I haven't modified any defaults.

It's on FreeBSD 6.1, and it's quite possible that it has something
wrong with utf-8 locale, I saw some complaints at least about
collation.

However, isspace doesn't seem like a culprit to me. Let's ensure that
I have done everything right to come to this conclusion. I have
applied the following patch to pl_funcs.c:

But, if I'm doing everything right, isspace seems not to be the
culprit to me. I have run the above query on the server with the
following applied patch (the server is 8.1.4, but I don't see any
changes to the plpgsql_convert_ident() in revs 1.46-1.54):

*** src/pl/plpgsql/src/pl_funcs.c.orig Wed Nov 23 00:23:30 2005
--- src/pl/plpgsql/src/pl_funcs.c Fri Sep 22 14:48:09 2006
***************
*** 363,368 ****
--- 363,381 ----
{
const char *sstart = s;
int identctr = 0;
+ char *isspacebuf;
+ int i;
+
+ isspacebuf = (char*)malloc(strlen(s) + 1);
+ for (i=0; s[i]; ++i) {
+ if (isspace(s[i]))
+ isspacebuf[i] = 't';
+ else
+ isspacebuf[i] = 'f';
+ }
+
+ ereport(LOG,
+ (errmsg("plpgsql_convert_ident: ident=%s, isspace array=%s, isspace(0)=%c", s, isspacebuf, isspace(0)?'t':'f')));

/* Outer loop over identifiers */
while (*s)

and got the following line in the logfile:

LOG: plpgsql_convert_ident: ident=х, isspace array=ff, isspace(0)=f

The symbol after "input=" is that 0x0445 (D1 85 in utf-8) letter which
I mentioned above. I also double-checked this: got a stack trace (by
inserting NULL-pointer assignment instead of reporting "expected dot
between identifiers") and saw that the value passed to the
plpgsql_convert_ident points to the following bytes: D1 85 00.

So, isspace returns false on all three of them.

I have also run the same query on another cluster (and the same
binaries), initialized with a single-byte encoding: ru_RU.KOI8-R,
where everything works. isspace returns the same results:

LOG: plpgsql_convert_ident: ident=х, isspace array=f, isspace(0)=f
LOG: plpgsql_convert_ident: ident=test_table, isspace array=ffffffffff, isspace(0)=f

Strange...

--
WBR, Victor V. Snezhko
E-mail: snezhko(at)indorsoft(dot)ru