Re: Performance monitoring

Jim C. Nasby wrote:
> Moving to -hackers.
>
> On Fri, May 11, 2007 at 04:37:44PM +0100, Heikki Linnakangas wrote:
>>> If you know when the checkpoint ended, and you know how long each of the
>>> pieces took, you can reconstruct the other times easily. The way you
>>> describe this it is true--that the summary is redundant given the
>>> detail--but if you put yourself in the shoes of a log file parser the
>>> other way around is easier to work with. Piecing together log entries
>>> is a pain, splitting them is easy.
>>>
>>> If I had to only keep one line out of this, it would be the one with the
>>> summary. It would be nice to have it logged at INFO.
>> Yeah, if we have the summary line we don't need the other lines and vice
>> versa. I have sympathy for parsing log files, I've done that a lot in
>> the past and I can see what you mean. Having the individual lines is
>> nice when you're monitoring a running system; you don't get the summary
>> line until the checkpoint is finished. I suppose we can have both the
>> individual lines and the summary, the extra lines shouldn't hurt anyone,
>> and you won't get them unless you turn on the new log_checkpoints
>> parameter anyway.
>
> Not to beat a dead horse, but do we really want to force folks to be
> parsing logs for performance monitoring? Especially if that log parsing
> is just going to result in data being inserted into a table anyway?
>
> I know there's concern about performance of the stats system and maybe
> that needs to be addressed, but pushing users to log parsing is a lot of
> extra effort, non-standard, likely to be overlooked, and doesn't play
> well with other tools. It also conflicts with all the existing
> statistics framework.

There is two counters for checkpoints in pgstats, the number of timed
(triggered by checkpoint_timeout) and requested (triggered by
checkpoint_segments) checkpoints.

Maybe we should improve the stats system so that we can collect events
with timestamps and durations, but in my experience log files actually
are the most reliable and universal way to collect real-time performance
information. Any serious tool has a generic log parser. The other
alternative is SNMP. I welcome the efforts on pgsnmpd..

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com