Re: Bad encoded chars in being inserted into database

We are working with 8.1 and migrating to 8.4....
We will see if after migration this behavior has disappeared. ;-)

Thank you, Scott.

-----Original Message-----
From: Scott Marlowe <scott(dot)marlowe(at)gmail(dot)com>
To: Iñigo Martinez Lasala <imartinez(at)vectorsf(dot)com>
Cc: pgsql-admin(at)postgresql(dot)org
Subject: Re: [ADMIN] Bad encoded chars in being inserted into database
Date: Mon, 22 Mar 2010 16:05:44 -0600

On Mon, Mar 22, 2010 at 2:48 AM, Iñigo Martinez Lasala
<imartinez(at)vectorsf(dot)com> wrote:
> Hi everybody.
>
>
>
> I have a doubt about how postgres deal with bad encoded characters into
> database.
>
> We have several gforge application. They are using postgres as database.
>
> If we export a database and import again, we have to deal with several bad
> encoded chars. These bad chars always come from copy & paste emails from
> Lotus Notes mail client. OK, I understand the Notes client people is using
> is an ancient application and does not deal very well with some Unicode
> chars…
>
> What I cannot understand is why postgres accept these bad enconded
> characters into database, exports them without problema but does not allow
> them when importing again.
>
> This has been happening since postgers 7.3. However, until 7.4.XX (y don’t
> remember what minor version) you could import database without ERRORs.
> However, since 7.4.XX it’s impossible and it’s imperative to clean bad
> characters (using iconv, for example) prior importing tables.

This is because postgresql's support for UTF-8 encoding (and all
encoding really) has gotten tighter over time, so that the filter to
catch improperly encoded UTF has gotten better with each major
release.