Skip site navigation (1) Skip section navigation (2)

Re: [HACKERS] Bad n_distinct estimation; hacks suggested?

From: Greg Stark <gsstark(at)mit(dot)edu>
To: "Dave Held" <dave(dot)held(at)arraysg(dot)com>
Cc: "pgsql-perform" <pgsql-performance(at)postgresql(dot)org>,<pgsql-hackers(at)postgresql(dot)org>
Subject: Re: [HACKERS] Bad n_distinct estimation; hacks suggested?
Date: 2005-04-27 17:16:48
Message-ID: 87ekcw3zsv.fsf@stark.xeocode.com (view raw or flat)
Thread:
Lists: pgsql-hackerspgsql-performance
"Dave Held" <dave(dot)held(at)arraysg(dot)com> writes:

> > Actually, it's more to characterize how large of a sample
> > we need.  For example, if we sample 0.005 of disk pages, and
> > get an estimate, and then sample another 0.005 of disk pages
> > and get an estimate which is not even close to the first
> > estimate, then we have an idea that this is a table which 
> > defies analysis based on small samples.  
> 
> I buy that.

Better yet is to use the entire sample you've gathered of .01 and then perform
analysis on that sample to see what the confidence interval is. Which is
effectively the same as what you're proposing except looking at every possible
partition.

Unfortunately the reality according to the papers that were sent earlier is
that you will always find the results disappointing. Until your sample is
nearly the entire table your estimates for n_distinct will be extremely
unreliable.

-- 
greg


In response to

Responses

pgsql-performance by date

Next:From: John A MeinelDate: 2005-04-27 17:20:37
Subject: Re: Final decision
Previous:From: Joshua D. DrakeDate: 2005-04-27 17:01:55
Subject: Re: Final decision

pgsql-hackers by date

Next:From: Bruce MomjianDate: 2005-04-27 17:37:38
Subject: Re: [HACKERS] Continue transactions after errors in psql
Previous:From: Tom LaneDate: 2005-04-27 17:16:40
Subject: Re: [HACKERS] Continue transactions after errors in psql

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group