From: | Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> |
---|---|
To: | Brar Piening <brar(at)gmx(dot)de> |
Cc: | Robert Haas <robertmhaas(at)gmail(dot)com>, Tatsuo Ishii <ishii(at)postgresql(dot)org>, david(at)kineticode(dot)com, itagaki(dot)takahiro(at)gmail(dot)com, pgsql-hackers(at)postgresql(dot)org |
Subject: | Re: Support UTF-8 files with BOM in COPY FROM |
Date: | 2011-09-26 19:48:53 |
Message-ID: | 29877.1317066533@sss.pgh.pa.us |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-hackers |
Brar Piening <brar(at)gmx(dot)de> writes:
> Citing from the Unicode FAQ again:
> Q: Where is a BOM useful?
> A: A BOM is useful at the beginning of files that are typed as text, but
> for which it is not known whether they are in big or little endian
> formatit can also serve as a hint indicating that the file is in
> Unicode, as opposed to in a legacy encoding and furthermore, it act as a
> signature for the specific encoding form used.
Note that the reference to byte order betrays the implicit context
assumption: that we're talking about UTF16 or UTF32 representation.
A BOM in UTF8 data is useless for its intended purpose of disambiguating
byte order. It could possibly be useful for telling UTF8 data apart
from non-UTF8 data, except for the inconvenient fact that that byte
sequence is not invalid data in non-UTF8 encodings.
BOM is useless in UTF8, no matter what Microsoft thinks. Any tool that
relies on it to detect UTF8 data has to have a workaround for overriding
that detection, or it's broken to the point of uselessness.
regards, tom lane
From | Date | Subject | |
---|---|---|---|
Next Message | Tom Lane | 2011-09-26 19:51:00 | Re: [v9.2] Fix Leaky View Problem |
Previous Message | Tom Lane | 2011-09-26 19:40:50 | Re: patch: plpgsql - remove unnecessary ccache search when a array variable is updated |