Chinese in Postgres

From: "ciifrancesco(at)tiscali(dot)it" <ciifrancesco(at)tiscali(dot)it>
To: pgsql-hackers(at)postgresql(dot)org, pgsql-zh-general(at)postgresql(dot)org, pgsql-ru-general(at)postgresql(dot)org
Subject: Chinese in Postgres
Date: 2013-08-16 11:25:31
Message-ID: 7171226.16231376652331594.JavaMail.defaultUser@defaultHost
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers pgsql-ru-general pgsql-zh-general

Hello all,
before writing this message, I wrote about this in other mailing lists without solving my problem.
Maybe some of you can help me.

I have problems with a DB in postgres, when i try to insert Chinese strings in UTF-8 format.
If I insert the data using a C++ program I have empty squares, in this format: ��� (3 empty squares for each chinese ideogram as that is the length in UTF-8)
If the string contains chinese mixed with ASCII, the ASCII is OK but the Chinese is broken:
漢語1-3漢語 --> ������1-3������

All the data is read from a binary file. It seems it's read correctly, but something happens when the query is executed.
(If the text is in a different language that uses only 2 bytes for each letter, I will see only 2 empty squares per character, ex. hebrew, but this is not good anyway...)

Strange things:
1. if i insert the record doing a query from command line (putty), the chinese text is OK. This problem is only when i insert by the C++ program.
2. I checked the C++ functions involved by creating unitary tests; if i run some other tests (on another virtual machine) the text is not damaged.
These strange things are confusing me, but maybe they will be useful informations for somebody who had the same problem.

The DB is set for UTF-8
Name | Owner | Encoding | Collate | Ctype | Access privileges
--------------+-------+----------+-------------+-------------+------------------
postgres | pgsql | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
MyDB | pgsql | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
template0 | pgsql | UTF8 | en_US.UTF-8 | en_US.UTF-8 |
template1 | pgsql | UTF8 | en_US.UTF-8 | en_US.UTF-8 |

Previously I also tried with:

Name | Owner | Encoding | Collate | Ctype | Access privileges
--------------+-------+----------+-------------+-------------+------------------
postgres | pgsql | UTF8 | C | C |
MyDB | pgsql | UTF8 | C | C |
...

But the problem was the same.
I know that you would like to see the code, but it's too long (anyway if you want i can try to write some lines of code, like connection to Db and so on). I don't know if there is some log create by postgres when inserting damaged data, sould be useful.

For now, in order to save your time my question is: did anybody of you have the same problem?
(and how did you solve it?)

Thanks,
Francesco

Invita i tuoi amici e Tiscali ti premia! Il consiglio di un amico vale più di uno spot in TV. Per ogni nuovo abbonato 30 € di premio per te e per lui! Un amico al mese e parli e navighi sempre gratis: http://freelosophy.tiscali.it/

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Hannu Krosing 2013-08-16 12:16:24 Re: Chinese in Postgres
Previous Message Rushabh Lathia 2013-08-16 06:22:26 Re: insert throw error when year field len > 4 for timestamptz datatype

Browse pgsql-ru-general by date

  From Date Subject
Next Message Hannu Krosing 2013-08-16 12:16:24 Re: Chinese in Postgres
Previous Message Alex Gorbachenko 2013-08-06 05:26:10 Re: VACUUM

Browse pgsql-zh-general by date

  From Date Subject
Next Message Hannu Krosing 2013-08-16 12:16:24 Re: Chinese in Postgres
Previous Message Florent Guillaume 2012-10-17 04:38:01 Re: WebSphere Application Server support for postgres