Re: Proposal: Adding json logging

On Mon, Apr 16, 2018 at 10:06:29AM -0400, Andrew Dunstan wrote:
> On 04/15/2018 05:05 PM, Christophe Pettus wrote:
> >> On Apr 15, 2018, at 12:16, David Arnold <dar(at)xoe(dot)solutions> wrote:
> >>
> >> Core-Problem: "Multi line logs are unnecessarily inconvenient to parse and are not compatible with the design of some (commonly used) logging aggregation flows."
> > I'd argue that the first line of attack on that should be to explain to those consumers of logs that they are making some unwarranted assumptions about the kind of inputs they'll be seeing. PostgreSQL's CSV log formats are not a particular bizarre format, or very difficult to parse. The standard Python CSV library handles them just file, for example. You have to handle newlines that are part of a log message somehow; a newline in a PostgreSQL query, for example, needs to be emitted to the logs.
>
>
> In JSON newlines would have to be escaped, since literal newlines are
> not legal in JSON strings. Postgres' own CSV parser has no difficulty at
> all in handling newlines embedded in the fields of CSV logs.

True, and anything that malloc()s in the process of doing that
escaping could fail on OOM, and hilarity would ensue. I don't see
these as show-stoppers, or even as super relevant to the vast majority
of users. If you're that close to the edge, you were going to crash
anyhow.

> I'm not necessarily opposed to providing for JSON logs, but the
> overhead of named keys could get substantial. Abbreviated keys might
> help, but generally I think I would want to put such logs on a
> compressed ZFS drive or some such.

Frequently at places I've worked, the end destination is of less
concern immediate than the ability to process those logs for
near-real-time monitoring. This is where formats like JSON really
shine.

Best,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate