Re: Possible rounding error of large float values?

On Tue, 2004-08-24 at 20:52, Josh Berkus wrote:
> Adam,
>
> > 9223372036854775807
> >
> > It gets selected out as:
> >
> > 9.22337203685478E18
>
> This is a property of FLOAT data types. They round.
>
> > Which appears to be rounded. When we cast it to numeric type we get:
> >
> > 9223372036854780000
> >
> > Which also is rounded. It is still possible to find the row using the
> > original value
>
> Hmmm ... is 15 digits the limit of NUMERIC? It may be.

It must be the limit of float. Numeric can't produce any more than it
was given and it was cast from float.

bray=# select 653596708775675750507850507570708696432 ::numeric;
numeric
-----------------------------------------
653596708775675750507850507570708696432
(1 row)

--
Oliver Elphick olly(at)lfix(dot)co(dot)uk
Isle of Wight http://www.lfix.co.uk/oliver
GPG: 1024D/A54310EA 92C8 39E7 280E 3631 3F0E 1EC0 5664 7A2F A543 10EA
========================================
"I saw in the night visions, and, behold, one like the
Son of man came with the clouds of heaven, and came to
the Ancient of days, and they brought him near before
him. And there was given him dominion, and glory, and
a kingdom, that all people, nations, and languages,
should serve him; his dominion is an everlasting
dominion, which shall not pass away, and his kingdom
that which shall not be destroyed."
Daniel 7:13,14