Skip site navigation (1) Skip section navigation (2)

Re: BUG #5711: input out of error with haversine formual

From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: "Vince" <vincecarney(at)gmail(dot)com>
Cc: pgsql-bugs(at)postgresql(dot)org
Subject: Re: BUG #5711: input out of error with haversine formual
Date: 2010-10-15 14:16:17
Message-ID: 19855.1287152177@sss.pgh.pa.us (view raw or flat)
Thread:
Lists: pgsql-bugs
"Vince" <vincecarney(at)gmail(dot)com> writes:
> If I break this down the following returns 1:
> SELECT (cos(radians(37.7438640)) * cos(radians(37.7438640)) *
> cos(radians(-97.4631299) - radians(-97.4631299)) + sin(radians(37.7438640))
> * sin(radians(37.743864000)));

No, it doesn't return 1, it returns 1-plus-a-little-bit, which is hidden
by float8out's desire to not print things like 1.0000000000000002.
Try it after setting extra_float_digits to 2 or so, or do

regression=# SELECT (cos(radians(37.7438640)) * cos(radians(37.7438640)) *
cos(radians(-97.4631299) - radians(-97.4631299)) + sin(radians(37.7438640))
* sin(radians(37.743864000))) - 1;
       ?column?       
----------------------
 2.22044604925031e-16
(1 row)

As somebody already remarked, you need to use a version of that formula
that's less prone to roundoff error.

			regards, tom lane

In response to

pgsql-bugs by date

Next:From: Magnus HaganderDate: 2010-10-15 15:03:11
Subject: Re: BUG #5687: RADIUS Authentication issues
Previous:From: Dean RasheedDate: 2010-10-15 07:23:24
Subject: Re: BUG #5711: input out of error with haversine formual

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group