Re: Possible rounding error of large float values?

On 8/24/2004 4:21 PM, Oliver Elphick wrote:
> On Tue, 2004-08-24 at 20:52, Josh Berkus wrote:
>> Adam,
>>
>> > 9223372036854775807
>> >
>> > It gets selected out as:
>> >
>> > 9.22337203685478E18
>>
>> This is a property of FLOAT data types. They round.
>>
>> > Which appears to be rounded. When we cast it to numeric type we get:
>> >
>> > 9223372036854780000
>> >
>> > Which also is rounded. It is still possible to find the row using the
>> > original value
>>
>> Hmmm ... is 15 digits the limit of NUMERIC? It may be.

The original numeric code I presented contained even log(), ln() and
trigonometric functions that produced results of up to 1,000 digits
precision. They wheren't considered usefull, so they didn't survive, but
numeric is that good after all.

Jan

>
> It must be the limit of float. Numeric can't produce any more than it
> was given and it was cast from float.
>
> bray=# select 653596708775675750507850507570708696432 ::numeric;
> numeric
> -----------------------------------------
> 653596708775675750507850507570708696432
> (1 row)
>

--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck(at)Yahoo(dot)com #