Re: auto_explain WAS: RFC: Timing Events

On Mon, Feb 25, 2013 at 8:26 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> On Sun, Feb 24, 2013 at 7:27 PM, Jim Nasby <jim(at)nasby(dot)net> wrote:
>> We actually do that in our application and have discovered that random
>> sampling can end up significantly skewing your data.
>
> /me blinks.
>
> How so?

Sampling is a pretty big area of statistics. There are dozens of
sampling methods to deal with various problems that occur with
different types of data distributions.

One problem is if you have some very rare events then random sampling
can produce odd results since those rare events will drop out entirely
unless your sample is very large whereas less rare events are
represented proportionally. There are sampling methods that ensure
that x% of the rare events are included even if those rare events are
less than x% of your total data set. One of those might be appropriate
to use for profiling data when you're looking for rare slow queries
amongst many faster queries.

--
greg