Re: Unicode problem again

On Tue, Jun 24, 2008 at 09:16:37AM +0200, Albe Laurenz wrote:
> Garry Saddington wrote:
> > ProgrammingError Error Value: ERROR: character 0xe28099 of
> > encoding "UTF8" has no equivalent in "LATIN1" select distinct
> [...]
>
> This is UNICODE 0x2019, a "right single quotation mark".
>
> This is a "Windows character" - the only non-UNICODE codepages I
> know that contain this character are the Microsoft codepages.
[...]
>
> > I have changed client_encoding to Latin1 to get over errors
> > caused by having the database in UTF8 and users trying to
> > enter special characters like £ signs.
> >
> > Unfortunately, it seems there are already UTF8 encodings in
> > the DB that have no equivalent in Latin1 from before the change.

Your input data seems to have a mix of encodings: sometimes you're
getting pound signs in a non-UTF-8 encoding, but if characters like
<U+2019 RIGHT SINGLE QUOTATION MARK> got into the database when
client_encoding was set to UTF8 then at least some data must have
been in UTF-8. If you're not certain that all data will be in the
same encoding then you might need to attempt to detect the encoding
and set client_encoding accordingly or convert the data to a common
encoding in the application before inserting it (I've had to do
this, sometimes on a line-by-line basis).

Setting client_encoding has implications for display as well as for
input: if the displaying application expects data in one encoding
but you give it data in a different encoding then non-ASCII characters
might not display correctly.

--
Michael Fuhr