Re: logging in high performance systems.

On 11/23/2011 09:28 PM, Theo Schlossnagle wrote:
> The first thing I did was add hook points where immediate statement
> logging happens "pre_exec" and those that present duration
> "post_exec". These should, with optimization turned on, have only a
> few instructions of impact when no hooks are registered (we could
> hoist the branch outside the function call if that were identified as
> an issue).
>
> https://github.com/postwait/postgres/commit/62bb9dfa2d373618f10e46678612720a3a01599a

Theo: could you e-mail the current "pre-flight and post-flight logging
hooks" code as a patch to the list? I put it into the CF app so we
don't forget to take a look at this, but from a policy point of view an
official patch submission to the list is needed here. I've looked at
the code on github enough to know we probably want the sort of
registration scheme you've proposed no matter what ends up getting logged.

The majority of Theo's patch is worrying about how to register and
manage multiple hook consumers around statements that are being
executed. The other one we got from Martin is hooking all log output
and not worrying about the multiple consumer problems.

There is an important distinction to make here. Statement logging is
one of the largest producers of logging data you want to worry about
optimizing for on a high performance system. The decision about whether
to log or not may need to be made by the hook. Something that hooks
EmitErrorReport has already lost an important opportunity to make a
decision about whether the system is perhaps too busy to worry about
logging right now at all; you've already paid a chunk of the logging
overhead getting the log message to it. I think both areas are going to
be important to hook eventually.

The meeting to discuss this area at our last BWPUG group happened, and
we made some good progress mapping out what various people would like to
see here. I'm holding the token for the job of summarizing all that
usefully for the community. I'll get back to that as soon as I can now
once the January CF is rolling along, hopefully later this week.

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com