Re: Proposal: Adding json logging

> I'd argue that the first line of attack on that should be to explain to
those consumers of logs that they are making some unwarranted assumptions
about the kind of inputs they'll be seeing. PostgreSQL's CSV log formats
are not a particular bizarre format, or very difficult to parse. The
standard Python CSV library handles them just file, for example. You have
to handle newlines that are part of a log message somehow; a newline in a
PostgreSQL query, for example, needs to be emitted to the logs.

I believe the (valid) root reason behind this assumption is to care for
resource consumption during stream processing. There are solutions to that
problem (as the multi line mode of fuent bit already exposed before in this
thread), but they are not specifically reliable, nor easy to maintain. Not
claiming this assumption does imply parsing of a *rolling* set of log lines
with *previously unkown cardinality*. That's expensive on computing
resources. I don't have actual numbers, but it doesn't seem too far
fetched, neither.

I filed a question to the author of fluent-bit to that extend which you can
consult here: https://github.com/fluent/fluent-bit/issues/564
Let's see what Eduardo has to inform us about this...

El dom., 15 abr. 2018 a las 16:05, Christophe Pettus (<xof(at)thebuild(dot)com>)
escribió:

>
> > On Apr 15, 2018, at 12:16, David Arnold <dar(at)xoe(dot)solutions> wrote:
> >
> > Core-Problem: "Multi line logs are unnecessarily inconvenient to parse
> and are not compatible with the design of some (commonly used) logging
> aggregation flows."
>
> I'd argue that the first line of attack on that should be to explain to
> those consumers of logs that they are making some unwarranted assumptions
> about the kind of inputs they'll be seeing. PostgreSQL's CSV log formats
> are not a particular bizarre format, or very difficult to parse. The
> standard Python CSV library handles them just file, for example. You have
> to handle newlines that are part of a log message somehow; a newline in a
> PostgreSQL query, for example, needs to be emitted to the logs.
>
> --
> -- Christophe Pettus
> xof(at)thebuild(dot)com
>
> --
[image: XOE Solutions] <http://xoe.solutions/> DAVID ARNOLD
Gerente General
xoe.solutions
dar(at)xoe(dot)solutions
+57 (315) 304 13 68
*Confidentiality Note: * This email may contain confidential and/or private
information. If you received this email in error please delete and notify
sender.
*Environmental Consideration: * Please avoid printing this email on paper,
unless really necessary.