Re: Floating point comparison inconsistencies of the geometric types

On Fri, May 27, 2016 at 6:43 AM, Emre Hasegeli <emre(at)hasegeli(dot)com> wrote:
> There are those macros defined for the built-in geometric types:
>
>> #define EPSILON 1.0E-06
>
>> #define FPzero(A) (fabs(A) <= EPSILON)
>> #define FPeq(A,B) (fabs((A) - (B)) <= EPSILON)
>> #define FPne(A,B) (fabs((A) - (B)) > EPSILON)
>> #define FPlt(A,B) ((B) - (A) > EPSILON)
>> #define FPle(A,B) ((A) - (B) <= EPSILON)
>> #define FPgt(A,B) ((A) - (B) > EPSILON)
>> #define FPge(A,B) ((B) - (A) <= EPSILON)
>
> with this warning:
>
>> * XXX These routines were not written by a numerical analyst.

I agree that those macros looks like a pile of suck. It's unclear to
me what purpose they're trying to accomplish, but regardless of what
it is, it's hard for me to believe that they are accomplishing it.
Whether 1.0E-06 is a correct fuzz factor presumably depends greatly on
the scale of the number; e.g. if the values are all like "3" it's
probably fine but if they are like "37142816124856" it's probably not
enough fuzz and if they are all like ".00000004" it's probably way too
much fuzz.

Figuring out what to do about it is harder. Your proposal seems to be
to remove them except where we need the fuzzy behavior, which doesn't
sound unreasonable, but I don't personally understand why we need it
in some places and not others. It would be good if some of the people
who are more numerically inclined than I am (and hate floats less, but
then that's everyone) could jump in here.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company