Re: rounding problems

On Wed, May 14, 2008 at 02:08:47PM -0400, Justin wrote:
> Sam Mason wrote:
> >What does foxpro use for storing numbers? or is it just that you never
> >pushed it hard enough for the abstractions to show through.
>
> I know i pushed it. Foxpro for the most has only 4 basic data types
> Numeric (similar to Posgresql numeric), Boolean, Date, Text aka
> (string) The foxpro tables supported far more data types but when every
> it was dumped to variable it acted like one of the 4.

I really meant how much did you check the results, or did you accept
that they were correct?

> Foxpro did not suffer floating point math errors. I normally used 8 to
> 10 points precision. Foxpro was limited to 15 points of precision
> period. No more and no less, once you hit that was it.

15 places seems very similar to what a 64bit IEEE floating point number
will give you, i.e. a double in C/C++.

> My problem is we calculate resistance of parts in a Foxpro app that we
> want to move because we want to bring all the custom apps into one
> framework and single database.
>
> Take this calculation (0.05/30000* 1.0025) which is used to calculate
> parts resistance and Tolerance. (its Ohms Law) The value returned from
> C++ = .0000016708 which is wrong
> it should be .00000167418. We just shrank the tolerance on the part we
> make

Why are you so sure about the FoxPro result? I've just checked a few
calculators and get results consistent with your C++ version.

Justin C: 0.0000016708
J FoxPro: 0.00000167418
My C: 0.000001670833
bc[1]: 0.0000016708333333333333333333333333333332
PG[2]: 0.0000016708333333333333336675
Google[3]: 0.00000167083333 (actually gives 1.67083333e-6)

Both bc and Postgres use their own code (i.e. not the CPU's FPU) to do
the math, and as they all agree I'm thinking FoxPro is incorrect! Next
I tried doing it accurately (in Haskell if it makes any difference) and
get an answer of 401/240000000 out, which would agree with everything
but FoxPro. If I calculate the ratio back out for FoxPro I get
401/239520242 which is a little way out.

> The Documentation from MS says 15 points of precision but the result say
> otherwise.

The docs for what? FoxPro or their C compiler?

If you mean FoxPro, I think this is another case of MS screwing up.

> I'm glad You and others are taking the time to explain to me
> the odd results before i get into redoing that application.

Welcome to the PG community, lots of people to get interested in lots of
things!

> Why oh Why did MS kill Foxpro. :'( I understood it, knew its quirks
> and it worked very well with Postgresql

Are you sure you want to stay with it if its answers are wrong?

Sam

[1] http://www.gnu.org/software/bc/manual/html_mono/bc.html
[2] http://doxygen.postgresql.org/backend_2utils_2adt_2numeric_8c-source.html
[3] http://www.google.com/search?q=0.05/30000*1.0025