Re: WIP - syslogger infrastructure changes

On Fri, Sep 25, 2009 at 22:26, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> On Fri, Sep 25, 2009 at 4:13 PM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>> On Fri, Sep 25, 2009 at 05:18, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>>> On Mon, Sep 14, 2009 at 4:14 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>>> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>>>>> First, the patch removes the logging_collector parameter and basically
>>>>> assumes that logging_collector is always on.
>>>>
>>>> I don't find that to be a good idea, and you certainly have not made
>>>> a case why we should change it.  I can't see any reason why pushing
>>>> functionality out of backends and downstream to the syslogger process
>>>> is an improvement.  What it's more likely to do is create a processing
>>>> bottleneck and a single point of failure.
>>>
>>> Hmm.  I think the justification was supposed to be this part here:
>>>
>>> $ With that, it's no longer necessary to restart your server just to
>>> $ reconfigure the logging, and it also takes away a confusing parameter
>>> $ (really "log_destination=stderr, logging_collector=on" is *not* a logical
>>> $ way to say "log to file"). Instead, it adds a log_destination of "file" that
>>> $ is the standard log to file."
>>>
>>> Do we have any positive or negative experience with logging_collector
>>> as a performance bottleneck?  Are there people running with
>>> logging_collector=off to avert disaster?
>>
>> I've never heard of that, so I'd be very interested in hearing if somebody did.
>>
>> Actually, it's not starting the logging collector that's the issue.
>> It's the redirection.
>>
>> So we could always start the logging collector, but not necessarily
>> redirect stderr. For those who then set logging to "file", it gets
>> passed to the logging collector. Those who set it to "stderr" just
>> have to deal with stderr somehow (pg_ctl -l for example). It's still
>> going to require a restart to change stderr, but that will only hit
>> those people who are actually having performance issues from it  (and
>> want to switch to using stderr instead of logging collector).
>>
>> Then we just send the data to the syslogger over a separate pipe. If
>> we keep the "endpoint" of this pipe in the postmaster, a new syslogger
>> could just "pick that up" after a crash, no? Data that goes on stderr
>> will still go to stderr and not be captured in this case, but normally
>> that would be no data at all, and if it did happen it'd be caught by
>> wherever you point pg_ctl -l at (which is the same place the early
>> startup messages goes, before the logger is running).
>
> That doesn't sound horribly robust.  I thought part of the point of
> the design of the logging collector was that if random libraries
> linked into the backend emit messages on stderr, we'll still capture
> them and send them on to the appropriate place, even though we don't
> really know what they are.  Having such messages go off to some
> probably-much-less-carefully-monitored logfile somewhere doesn't seem
> good.

Well, from how I read the first complaint here, running with
logging_collector on is simply broken from the crash perspective, no?
I mean, don't we have the same problem *today*, it's just that
logging_collector isn't on by default (in the source distribution that
is - on our most installed platform it is..)

Also, how often does it actually happen that "random libraries linked
into the backend emit messages on stderr"? Is this really something
that we need to pay so much attention to, rather than just let them go
on stderr and have them be redirected?

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/