Skip site navigation (1) Skip section navigation (2)

Re: logging in high performance systems.

From: Martin Pihlak <martin(dot)pihlak(at)gmail(dot)com>
To: Theo Schlossnagle <jesus(at)omniti(dot)com>
Cc: pgsql-hackers(at)postgresql(dot)org
Subject: Re: logging in high performance systems.
Date: 2011-12-04 22:10:03
Message-ID: 4EDBEFBB.9070509@gmail.com (view raw or flat)
Thread:
Lists: pgsql-hackers
On 11/24/2011 06:33 PM, Theo Schlossnagle wrote:
> I see the next steps being:
>  1) agreeing that a problem exists (I know one does, but I suppose
> consensus is req'd)

+1, high volume logging is also a huge problem here at Skype. Some of
the issues that immediately come to mind:

- extracting usage statistics from logs
- extracting error messages from logs
- pushing the logs to remote log server (and no, syslog is not actually
  usable for that ;)

>  2) agreeing that "hooks" are the right approach, if not propose a
> different approach. (fwiw, it's incredible common)

+1 for hook based approach. While the whole logging system could
possibly use an overhaul, this is not likely going to happen for 9.2.
Meanwhile hooks provide the flexibility to implement custom log
collectors for those who really need it.

>  3) reworking the implementation to fit in the project; I assume the
> implementation I proposed will, at best, vaguely resemble anything
> that gets integrated. It was just a PoC.
> 

I'd vote for a hook to be added to EmitErrorReport. That way all logged
messages pass through the hook, which could then decide whether to
actually write the message to server log or to discard it. As an added
bonus, the message is  broken down into pieces so there is no need to
parse the messages for severity, context, etc.

Patch attached.

regards,
Martin

Attachment: emit-log-hook.patch
Description: text/x-patch (1.4 KB)

In response to

pgsql-hackers by date

Next:From: Tom LaneDate: 2011-12-04 22:11:07
Subject: Re: planner fails on HEAD
Previous:From: Tomas VondraDate: 2011-12-04 21:56:20
Subject: Re: why local_preload_libraries does require a separate directory ?

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group