Skip site navigation (1) Skip section navigation (2)

logging in high performance systems.

From: Theo Schlossnagle <jesus(at)omniti(dot)com>
To: pgsql-hackers(at)postgresql(dot)org
Subject: logging in high performance systems.
Date: 2011-11-24 02:28:06
Message-ID: (view raw or whole thread)
Lists: pgsql-hackers
We have a need for logging in systems where it isn't feasible to log
to disk as it negatively impacts performance.

I'd like to be able to creatively solve this problem without modifying
the core, but today I cannot.

So... here's my first whack at solving this with some flexibility.

The first thing I did was add hook points where immediate statement
logging happens "pre_exec" and those that present duration
"post_exec".  These should, with optimization turned on, have only a
few instructions of impact when no hooks are registered (we could
hoist the branch outside the function call if that were identified as
an issue).

The second thing I did was write a sample use of those hooks to
implement a completely non-blocking fifo logger. (if it would block,
it drops the log line).  The concept is that we could run this without
risk of negative performance impact due to slow log reading (choosing
to drop logs in lieu of pausing).  And a simple process could be
written to consume from the fifo.  We use this method in other systems
to log many 10s of thousands of log lines per second with negligible
impact on performance.

Thoughts? Feedback?

As can be seen, the patch is pretty tiny.

Theo Schlossnagle


pgsql-hackers by date

Next:From: Tom LaneDate: 2011-11-24 03:33:19
Subject: Obstacles to user-defined range canonicalization functions
Previous:From: Kevin GrittnerDate: 2011-11-24 00:36:02
Subject: Re: [JDBC] Optimize postgres protocol for fixed size arrays

Privacy Policy | About PostgreSQL
Copyright © 1996-2015 The PostgreSQL Global Development Group