Re: my float function returns NaN

From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Nathaniel Trellice <naptrel(at)yahoo(dot)co(dot)uk>
Cc: Frank Bax <fbax(at)sympatico(dot)ca>, "pgsql-novice(at)postgresql(dot)org" <pgsql-novice(at)postgresql(dot)org>
Subject: Re: my float function returns NaN
Date: 2012-04-07 14:54:22
Message-ID: 2542.1333810462@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-novice

Nathaniel Trellice <naptrel(at)yahoo(dot)co(dot)uk> writes:
> My guess would be that due to rounding errors, the argument passed to
> acos() is outwith the interval [-1,1] and acos() is only defined
> within this interval (for reals).

Yeah. It's not helping any that you've defined geo_deg2rad() to do its
arithmetic in float (that is float4) precision. Making it work in
float8 would probably improve matters. Still, I agree that switching to
a more numerically stable distance calculation would be a good idea.

regards, tom lane

In response to

Responses

Browse pgsql-novice by date

  From Date Subject
Next Message Frank Bax 2012-04-07 16:24:56 Re: my float function returns NaN
Previous Message Nathaniel Trellice 2012-04-07 08:32:35 Re: my float function returns NaN