Skip site navigation (1) Skip section navigation (2)

Re: logging in high performance systems.

From: Greg Smith <greg(at)2ndQuadrant(dot)com>
To: pgsql-hackers(at)postgresql(dot)org, jesus(at)omniti(dot)com
Subject: Re: logging in high performance systems.
Date: 2012-01-16 08:51:21
Message-ID: 4F13E509.1070907@2ndQuadrant.com (view raw or flat)
Thread:
Lists: pgsql-hackers
On 11/23/2011 09:28 PM, Theo Schlossnagle wrote:
> The first thing I did was add hook points where immediate statement
> logging happens "pre_exec" and those that present duration
> "post_exec".  These should, with optimization turned on, have only a
> few instructions of impact when no hooks are registered (we could
> hoist the branch outside the function call if that were identified as
> an issue).
>
> https://github.com/postwait/postgres/commit/62bb9dfa2d373618f10e46678612720a3a01599a

Theo:  could you e-mail the current "pre-flight and post-flight logging 
hooks" code as a patch to the list?  I put it into the CF app so we 
don't forget to take a look at this, but from a policy point of view an 
official patch submission to the list is needed here.  I've looked at 
the code on github enough to know we probably want the sort of 
registration scheme you've proposed no matter what ends up getting logged.

The majority of Theo's patch is worrying about how to register and 
manage multiple hook consumers around statements that are being 
executed.  The other one we got from Martin is hooking all log output 
and not worrying about the multiple consumer problems.

There is an important distinction to make here.  Statement logging is 
one of the largest producers of logging data you want to worry about 
optimizing for on a high performance system.  The decision about whether 
to log or not may need to be made by the hook.  Something that hooks 
EmitErrorReport has already lost an important opportunity to make a 
decision about whether the system is perhaps too busy to worry about 
logging right now at all; you've already paid a chunk of the logging 
overhead getting the log message to it.  I think both areas are going to 
be important to hook eventually.

The meeting to discuss this area at our last BWPUG group happened, and 
we made some good progress mapping out what various people would like to 
see here.  I'm holding the token for the job of summarizing all that 
usefully for the community.  I'll get back to that as soon as I can now 
once the January CF is rolling along, hopefully later this week.

-- 
Greg Smith   2ndQuadrant US    greg(at)2ndQuadrant(dot)com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com


In response to

Responses

pgsql-hackers by date

Next:From: Simon RiggsDate: 2012-01-16 08:51:45
Subject: Re: SKIP LOCKED DATA
Previous:From: Fujii MasaoDate: 2012-01-16 08:17:31
Subject: Re: Standalone synchronous master

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group