Re: Generic Monitoring Framework Proposal

On Jun 19, 2006, at 4:40 PM, Tom Lane wrote:

> Robert Lor <Robert(dot)Lor(at)Sun(dot)COM> writes:
>> The main goal for this Generic Monitoring Framework is to provide a
>> common interface for adding instrumentation points or probes to
>> Postgres so its behavior can be easily observed by developers and
>> administrators even in production systems.
>
> What is the overhead of a "probe" when you're not using it? The
> answer
> had better not include the phrase "kernel call", or this is
> unlikely to
> pass muster...
>
>> For DTrace, probes can be enabled using a D script. When the
>> probes are not enabled, there is absolutely no performance hit
>> whatsoever.
>
> If you believe that, I have a bridge in Brooklyn you might be
> interested
> in.

Heh. Syscall probes and FBT probes in Dtrace have zero overhead.
User-space probes do have overhead, but it is only a few instructions
(two I think). Besically, the probe points are replaced by illegal
instructions and the kernel infrastructure for Dtrace will fasttrap
the ops and then act. So, it is tiny tiny overhead. Little enough
that it isn't unreasonable to instrument things like s_lock which are
tiny.

> What are the criteria going to be for where to put probe calls? If it
> has to be hard-wired into the source code, I foresee a lot of
> contention
> about which probes are worth their overhead, because we'll need
> one-size-fits-all answers.
>
>> arg1..arg5 = Any args to pass to the probe such as txn id, lock
>> id, etc
> Where is the data type of a probe argument defined?

I assume it would depend on the probe implementation. In Dtrace they
are implemented in .d files that will post-instrument the object
before final linkage. Dtrace's whole purpose is to be low overhead
and it really does it in a fantastic way.

As an example, you can take an uninstrumented binary and add dynamic
instrumentation to the entry, exit and every instruction op-code over
every single routine in the process. And clearly, as the binary is
uninstrumented, the overhead is indeed zero when the probes are not
enabled.

The reason that Robert proposes user-space probes (I assume) is that
tracing C functions can be too granular and not conveniently expose
the "right" information to make tracing useful.

// Theo Schlossnagle
// CTO -- http://www.omniti.com/~jesus/
// OmniTI Computer Consulting, Inc. -- http://www.omniti.com/
// Ecelerity: Run with it.