On 12/8/05, Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> wrote:
> > A script which identifies non-utf-8 characters and provides some
> > context, line numbers, etc, will greatly speed up the process of
> > remedying the situation.
>
> I think the best we can do is the "iconv -c with the diff" idea, which
> is already in the release notes. I suppose we could merge the iconv and
> diff into a single command, but I don't see a portable way to output the
> iconv output to stdout., /dev/stdin not being portable.
No, what is needed for people who care about fixing their data is a
loadable strip_invalid_utf8() that works in older versions.. then just
select * from bar where foo != strip_invalid_utf8(foo); The function
would be useful in general, for example, if you have an application
which doesn't already have much utf8 logic, you want to use a text
field, and stripping is the behaviour you want. For example, lots of
simple web applications.