Re: [HACKERS] again: Bug #943: Server-Encoding from EUC_TW toUTF-8

Tatsuo Ishii wrote:
>
> > Hello,
> > I reported bug #943 (I found in 7.3.2) and you checked in some change against integer overflow.
> > Now I upgraded to 7.3.3 and I'm not happy with this.
> > The exact error as I described is fixed, but I found new errors in conversion UTF-8 <-> EUC_TW and BIG5:
> >
> > Copy to table (DB has UTF-8 encoding) from file:
> > for PGCLIENTENCODING=BIG5:
> > WARNING: copy: line 1, LocalToUtf: could not convert (0xf9d6) BIG5 to UTF-8. Ignored
> > WARNING: copy: line 2, LocalToUtf: could not convert (0xf9d7) BIG5 to UTF-8. Ignored
> > WARNING: copy: line 3, LocalToUtf: could not convert (0xf9d8) BIG5 to UTF-8. Ignored
> > WARNING: copy: line 4, LocalToUtf: could not convert (0xf9db) BIG5 to UTF-8. Ignored
>
> I see no problem here. The only standard conversion map I could found
> on-line form so far (see below URL) does not include entries 0xf9d6 or
> above.

Sorry, I do not know anything about conversion maps and CNS 11643-1993 planes.
I only got a file in BIG5 encoding from Taiwan and found that it is not possible
to load all text to postgresql 7.3.3.
But it is possible to convert to UTF-8 with iconv tool from glibc (Linux).
It would be good if next release supports todays BIG5.

Michael

> http://www.unicode.org/Public/UNIDATA/Unihan.txt
>
> > for EUC_TW
> > WARNING: copy: line 1, LocalToUtf: could not convert (0x8ea3c3b7) EUC_TW to UTF-8. Ignored
> > WARNING: copy: line 2, LocalToUtf: could not convert (0x8ea3cfd0) EUC_TW to UTF-8. Ignored
> > WARNING: copy: line 3, LocalToUtf: could not convert (0x8ea3c4ce) EUC_TW to UTF-8. Ignored
> > WARNING: copy: line 4, LocalToUtf: could not convert (0x8ea3bdfe) EUC_TW to UTF-8. Ignored
>
> Hum. These seem to be CNS 11643-1993, plane 3. Currently PostgreSQL
> supports only:
>
> CNS 11643-1993, plane 0
> CNS 11643-1993, plane 1
> CNS 11643-1993, plane 2
> CNS 11643-1993, plane 15
>
> Would you like to have support for rest of CNS 11643-1993 planes:
>
> CNS 11643-1993, plane 3
> CNS 11643-1993, plane 4
> CNS 11643-1993, plane 5
> CNS 11643-1993, plane 6
> CNS 11643-1993, plane 7
>
> support for upcoming 7.4?
>
> > Copy out to file from table (UTF-8 data):
> > to BIG5
> > WARNING: UtfToLocal: could not convert UTF-8 (0xe7a281). Ignored
> > WARNING: UtfToLocal: could not convert UTF-8 (0xe98ab9). Ignored
> > WARNING: UtfToLocal: could not convert UTF-8 (0xe8a38f). Ignored
> > WARNING: UtfToLocal: could not convert UTF-8 (0xe7b2a7). Ignored
> >
> > to EUC_TW is ok!
>
> BIG5 and EUC_TW have different code points. So this is not very strange.
> --
> Tatsuo Ishii