Re: Possible rounding error of large float values?

# select 9223372036854775807 = 9223372036854775807::float;
?column?
----------
t
(1 row)

# select 9223372036854775807 = 9223372036854775807.0::float;
?column?
----------
t
(1 row)

# select 9223372036854775807 = 9223372036854775807.0000::numeric;
?column?
----------
t
(1 row)

This and the fact that it's still possible to find the row using the
original value would seem to indicate that the rounding is just a display
artifact..

----- Original Message -----
From: "Oliver Elphick" <olly(at)lfix(dot)co(dot)uk>
To: "Josh Berkus" <josh(at)agliodbs(dot)com>
Cc: "Adam Lancaster" <alancaster(at)preventsys(dot)com>; <pgsql-sql(at)postgresql(dot)org>
Sent: Wednesday, August 25, 2004 5:21 AM
Subject: Re: [SQL] Possible rounding error of large float values?

> On Tue, 2004-08-24 at 20:52, Josh Berkus wrote:
> > Adam,
> >
> > > 9223372036854775807
> > >
> > > It gets selected out as:
> > >
> > > 9.22337203685478E18
> >
> > This is a property of FLOAT data types. They round.
> >
> > > Which appears to be rounded. When we cast it to numeric type we get:
> > >
> > > 9223372036854780000
> > >
> > > Which also is rounded. It is still possible to find the row using the
> > > original value
> >
> > Hmmm ... is 15 digits the limit of NUMERIC? It may be.
>
> It must be the limit of float. Numeric can't produce any more than it
> was given and it was cast from float.
>
> bray=# select 653596708775675750507850507570708696432 ::numeric;
> numeric
> -----------------------------------------
> 653596708775675750507850507570708696432
> (1 row)
>
> --
> Oliver Elphick olly(at)lfix(dot)co(dot)uk
> Isle of Wight http://www.lfix.co.uk/oliver
> GPG: 1024D/A54310EA 92C8 39E7 280E 3631 3F0E 1EC0 5664 7A2F A543 10EA
> ========================================
> "I saw in the night visions, and, behold, one like the
> Son of man came with the clouds of heaven, and came to
> the Ancient of days, and they brought him near before
> him. And there was given him dominion, and glory, and
> a kingdom, that all people, nations, and languages,
> should serve him; his dominion is an everlasting
> dominion, which shall not pass away, and his kingdom
> that which shall not be destroyed."
> Daniel 7:13,14
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster