Re: Proposal: Adding json logging

>Why? The newlines aren't meaningfully different from other characters
you need to parse? The data isn't actually stored in a newline separated
fashion, that's just one byte with that meaning

I miss the details, but I believe that stdout is usually parsed and
streamed simply line by line. Like in:
"Important: do not attempt to add multiline support in your regular
expressions if you are using Tail
<https://fluentbit.io/documentation/0.13/input/tail.html> input plugin
since each line is handled as a separated entity. Instead use Tail Multiline
<https://fluentbit.io/documentation/0.13/input/tail.html#multiline> support
configuration feature."
https://fluentbit.io/documentation/0.13/parser/regular_expression.html

El dom., 15 abr. 2018 a las 13:08, David Arnold (<dar(at)xoe(dot)solutions>)
escribió:

> >This would appear to solve multiline issues within Fluent.....
> >https://docs.fluentd.org/v0.12/articles/parser_multiline
>
> I definitely looked at that, but what guarantees do I have that the
> sequence is always ERROR/STATEMENT/DETAIL? And not the other way round?
> And it only works with tail logging from log file so I cannot use a native
> docker logging driver which streams event by event.
> This again prohibits me the usage of host-global docker logging driver
> configuration as my standard option for host provisioning.
>
> >Is the issue that there are line breaks in things like lines 7-9?
> No way to parse that with a line by line regex cleanly. Fluent-multi line
> is no real multi line regex, it's just some logic to emulate multi line
> regex. I believe the reason might be that true multi line regex would be
> way to resource demanding to run intercalatingly on a moving set of lines
> of unknown cardinality.
>
> El dom., 15 abr. 2018 a las 13:00, David Arnold (<dar(at)xoe(dot)solutions>)
> escribió:
>
>> >It looks like the thread skipped over the problem space for the
>> solution space pretty fast
>>
>> OK, I apologize, it seemed to me from the feedback that the problem was
>> already uncontested. To verify/falsify that was the objective of my
>> previous mail :)
>>
>> >Can you elaborate?
>>
>> Sure.
>> CSV Logs: https://pastebin.com/uwfmRdU7
>> stdout:
>> https://gist.github.com/blaggacao/d389940ef7b223eebc707096b536967d
>>
>> CSV shows line breaks, STDOUT shows ERROR/FATAL and detail on different
>> lines, not an easy problem to stream-parse reliably (without some kind of a
>> buffer, etc)...
>>
>> El dom., 15 abr. 2018 a las 12:46, Christophe Pettus (<xof(at)thebuild(dot)com>)
>> escribió:
>>
>>>
>>> > On Apr 15, 2018, at 10:39, David Arnold <dar(at)xoe(dot)solutions> wrote:
>>> >
>>> > In the light of the specific use case / problem for this thread to be
>>> born, what exactly would you suggest?
>>>
>>> It looks like the thread skipped over the problem space for the solution
>>> space pretty fast; I see your note:
>>>
>>> > I have reviewed some log samples and all DO contain some kind of multi
>>> line logs which are very uncomfortable to parse reliably in a log streamer.
>>>
>>> ... but I don't see any actual examples of those. Can you elaborate?
>>>
>>> --
>>> -- Christophe Pettus
>>> xof(at)thebuild(dot)com
>>>
>>> --
>> [image: XOE Solutions] <http://xoe.solutions/> DAVID ARNOLD
>> Gerente General
>> xoe.solutions
>> dar(at)xoe(dot)solutions
>> +57 (315) 304 13 68
>> *Confidentiality Note: * This email may contain confidential and/or
>> private information. If you received this email in error please delete and
>> notify sender.
>> *Environmental Consideration: * Please avoid printing this email on
>> paper, unless really necessary.
>>
> --
> [image: XOE Solutions] <http://xoe.solutions/> DAVID ARNOLD
> Gerente General
> xoe.solutions
> dar(at)xoe(dot)solutions
> +57 (315) 304 13 68
> *Confidentiality Note: * This email may contain confidential and/or
> private information. If you received this email in error please delete and
> notify sender.
> *Environmental Consideration: * Please avoid printing this email on
> paper, unless really necessary.
>
--
[image: XOE Solutions] <http://xoe.solutions/> DAVID ARNOLD
Gerente General
xoe.solutions
dar(at)xoe(dot)solutions
+57 (315) 304 13 68
*Confidentiality Note: * This email may contain confidential and/or private
information. If you received this email in error please delete and notify
sender.
*Environmental Consideration: * Please avoid printing this email on paper,
unless really necessary.