Re: BUG #5661: The character encoding in logfile is confusing.

On 09/22/2010 09:55 PM, Tom Lane wrote:
> Peter Eisentraut<peter_e(at)gmx(dot)net> writes:
>> On ons, 2010-09-22 at 19:25 +0800, Craig Ringer wrote:
>>> I still wonder if, rather than making this configurable, the right
>>> choice is to force logging to UTF-8 (with BOM) across the board,
>
>> I don't think this would make things better or easier. At some point
>> you're going to have to insert a recode call, and it doesn't matter much
>> whether the destination argument is a constant or a variable.
>
> It'd avoid the problem of having possibly-unconvertable messages ...
> at the cost of pissing off users who have a uniform server encoding
> selection already and don't see why they should be forced to deal with
> UTF8 in the log.
>
> It's pretty much just one step from here to deciding that the server
> should work exclusively in UTF8 and never mind all those other legacy
> encodings. We've resisted that attitude for quite some years now,
> and are probably not really ready to adopt it for the log either.

Fair enough. The current approach is broken, though. Mis-encoded
messages the user can't read are little more good to them than messages
that're never logged.

I see four options here (two of which are practical IMO):

(1) Log in UTF-8, convert everything to UTF-8. Better for admin tools &
apps, sucks for OS utilities/grep/etc on non-utf-8 locales. Preserves
all messages no matter what the database and system encodings are.

(2) Log in default encoding for locale, convert all messages to that
encoding. Where characters cannot be represented in the target encoding
replace them with a placeholder (? or something). Better - but far from
good - for OS utilities/grep/etc, sucks for admin tools and apps.
Doesn't preserve all messages properly if user has databases in
encodings other than the system encoding.

(3) Have a log for the postmaster in the default locale for the system.
Have a log file for each database that's in the encoding for that
database. IMO this is the worst of both worlds, but it does preserve
original encodings without transcoding or forcing a particular encoding
and does preserve messages. Horribly complicated for admin tools,
inconsistent and horrid for grep etc.

(4) Keep things much as they are, but log an encoding identifier prefix
for each line. Lets GUI/admin tools post-process the logs into something
sane, permits automated log processing because line encodings are known.
Sucks for shell tools, which can't tell which lines are which; we'd need
to provide a "pggrep" and "pgless" for reliable log search! Preserves
all messages, but not in a reliably searchable manner.

(0) Change nothing. Log all messages in the original encoding they were
generated in. Perform no conversion. Logs contain mixed encodings.
Horrible for admin/gui tools (broken text). Horrible for shell
utilities/OS tools (can't trust grep results etc). Automatic log
processing impossible as the encoding for each line isn't known and
can't be reliably discovered.

As far as I'm concerned, (3) is out. It's horrible. I don't think the
status quo (0) is OK either, it's producing broken log files. (4) is
pretty awful too, but it's the smallest change that kind-of fixes the
issue to the point where it's at least possible for PgAdmin etc to
convert the logs into a consistent encoding.

IMO it's down to (1) and (2). There's no clear consensus between those
two, so I'd be inclined to offer the admin the choice between them as a
config option, depending on the trade-off they prefer to make.

For sensible systems in a utf-8 locale (1) and (2) are equivalent, and
(2) is fine for systems where the database encoding is always the same
as the system encoding. It's only for systems with a non-utf-8 locale
that use databases in encodings other than the system locale's encoding
that problems arise. In this case they're going to get suboptimal
results one way or the other, it's just a matter of letting them pick how.

Thoughts?

I should ask on the various language-specific mailing lists and see what
people there have to say about it. Maybe it doesn't affect people enough
in practice for them to care.

--
Craig Ringer