Re: [PATCH] Log CSV by default

From: Gilles Darold <gilles(at)darold(dot)net>
To: David Fetter <david(at)fetter(dot)org>, Gilles Darold <gilles(at)darold(dot)net>
Cc: PostgreSQL Development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: [PATCH] Log CSV by default
Date: 2018-12-03 20:20:01
Message-ID: 17cccd7d-c66d-cd30-6b67-c34a72c14143@darold.net
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Le 03/12/2018 à 19:25, David Fetter a écrit :
> On Mon, Dec 03, 2018 at 07:18:31PM +0100, Gilles Darold wrote:
>> Le 30/11/2018 à 19:53, David Fetter a écrit :
>>> This makes it much simpler for computers to use the logs while not
>>> making it excessively difficult for humans to use them.
>> I'm also not very comfortable with this change. For a pgBadger point
>> of view I don't know an existing library to parse csv log in
>> multi-process mode. The only way I know is to split the file or load
>> chunks in memory which is not really recommended with huge log
>> files. I am not saying that this is not possible to have a Perl CSV
>> library allowing multi-process but this require some additional days
>> of work.
> How about other structured log formats? Would one for JSON work
> better?

pgBadger is able to parse jsonlog
(https://github.com/michaelpq/pg_plugins/tree/master/jsonlog) output in
multi-process mode because it do not use an external library to parse
the json. In this minimalist implementation I assume that each log line
is a full json object. Honestly I do not have enough feedback on this
format to be sure that my basic jsonlog parser is enough, maybe it is
not. What is really annoying for a log parser is the preservation of
newline character in queries, if the format outputs each atomic log
messages on a single line each, this is ideal.

>
>> Also at this time I have not received any feature request for that,
>> maybe the use of csvlog format is not so common.
> Part of the reason it's uncommon is that it's not the default we ship
> with, and defaults influence this very much.

I agree. I also think that syslog or stderr are more human readable than
csvlog or jsonlog, which have probably contributed to their large
adoption too.

--
Gilles Darold
http://www.darold.net/

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Stephen Frost 2018-12-03 21:20:47 Re: [PATCH v19] GSSAPI encryption support
Previous Message Pavel Stehule 2018-12-03 19:16:37 citelny zajimavy clanek venovany fyzicke replikaci