Re: Bug in UTF8-Validation Code?

Mark Dilger wrote:
> Bruce Momjian wrote:
>> Added to TODO:
>>
>> * Fix cases where invalid byte encodings are accepted by the
>> database,
>> but throw an error on SELECT
>>
>> http://archives.postgresql.org/pgsql-hackers/2007-03/msg00767.php
>>
>> Is anyone working on fixing this bug?
>
> Hi, has anyone volunteered to fix this bug? I did not see any reply on
> the mailing list to your question above.
>
> mark

OK, I can take a stab at fixing this. I'd like to state some assumptions so
people can comment and reply:

I assume that I need to fix *all* cases where invalid byte encodings get into
the database through functions shipped in the core distribution.

I assume I do not need to worry about people getting bad data into the system
through their own database extensions.

I assume that the COPY problem discussed up-thread goes away once you eliminate
all the paths by which bad data can get into the system. However, existing
database installations with bad data already loaded will not be magically fixed
with these code patches.

Do any of the string functions (see
http://www.postgresql.org/docs/8.2/interactive/functions-string.html) run the
risk of generating invalid utf8 encoded strings? Do I need to add checks? Are
there known bugs with these functions in this regard?

If not, I assume I can add mbverify calls to the various input routines (textin,
varcharin, etc) where invalid utf8 could otherwise enter the system.

I assume that this work can be limited to HEAD and that I don't need to
back-patch it. (I suspect this assumption is a contentious one.)

Advice and comments are welcome,

mark