Re: syslog_line_prefix

On Mon, Sep 28, 2009 at 5:22 AM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> On Sun, Sep 27, 2009 at 23:03, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>> On Sun, Sep 27, 2009 at 4:54 PM, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:
>>> On Sun, 2009-09-27 at 16:15 -0400, Tom Lane wrote:
>>>> Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
>>>> > Then why not send everything to syslog and have syslog filter it to the
>>>> > places you want to?  That is what syslog is for, after all.
>>>>
>>>> We send all syslog output with the same identifier/priority/facility,
>>>> so there's not a lot of hope of getting syslog to do any useful
>>>> filtering (at least not with the versions of syslog I'm familiar with).
>>>
>>> Time to upgrade then. ;-)  For example, the default syslog in Fedora has
>>> been rsyslog since Fedora 8, and that one can do a lot more than just
>>> filter by identifier/priority/facility.  syslog-ng is another popular
>>> example for a more featureful syslog.
>>
>> Presumably csvlog would be good for these sorts of things too, no?
>> The whole point is it's machine-readable.
>
> If there was a way to pipe the csv log through an external problem,
> that would take care of much of the problem.

tail -f is probably a bit too fragile for this purpose, but I think it
would be possible to design a utility that would do this. The idea
would be to maintain a state file that would list the most recent CSV
log file read and the byte offset of the first byte following the last
line processed. On every iteration, we just open up the last file
read and read beginning at the designated offset through end of file.
Then we check if a newer file is available and, if so, we begin
reading that file. When we're done reading, we update the state file.

There is the problem of what happens if we read a partial last line of
a file being written, but that seems surmountable: if the last line
read does not end in a newline, and no newer file is present, then
don't include that partial line in the output, and record the offset
of the beginning of that line in the state file.

I'm not sure if this will work on Windows, but it should be OK on
anything UNIX-ish.

> And I guess if you make that program responsible for *everything* it
> would work - you just set your logging level to log very much data,
> and let the external process deal with it. If we implemented the
> ability to have a different logging level for different destinations
> you could keep text logging for other things, or you could just
> delegate all that to the external process as well. That would
> basically turn the syslogger into a process that reads from the input
> and sends the data out to an external process. But it could then
> implement things like automatic restart of the external process in
> case of crash etc, in perhaps a much easier way than the postmaster
> can do for the syslogger itself.

The problem with having the syslogger send the data directly to an
external process is that the external process might be unable to
process the data as fast as syslogger is sending it. I'm not sure
exactly what will happen in that case, but it will definitely be bad.
I think what will likely happen is that the entire database cluster
will end up waiting on write(2) calls to various places and processing
will grind to a halt.

I think it's better to spool the log messages to files, and then let
the external utility read the files. The external utility can still
fall behind, but even if it does the cluster will continue running.

...Robert