Skip site navigation (1) Skip section navigation (2)

Re: Possible rounding error of large float values?

From: Oliver Elphick <olly(at)lfix(dot)co(dot)uk>
To: Josh Berkus <josh(at)agliodbs(dot)com>
Cc: Adam Lancaster <alancaster(at)preventsys(dot)com>,pgsql-sql(at)postgresql(dot)org
Subject: Re: Possible rounding error of large float values?
Date: 2004-08-24 20:21:15
Message-ID: 1093378875.15248.39.camel@linda (view raw or flat)
Thread:
Lists: pgsql-sql
On Tue, 2004-08-24 at 20:52, Josh Berkus wrote:
> Adam,
> 
> > 9223372036854775807
> >
> > It gets selected out as:
> >
> > 9.22337203685478E18
> 
> This is a property of FLOAT data types.  They round.
> 
> > Which appears to be rounded. When we cast it to numeric type we get:
> >
> > 9223372036854780000
> >
> > Which also is rounded. It is still possible to find the row using the
> > original value
> 
> Hmmm ... is 15 digits the limit of NUMERIC?   It may be.

It must be the limit of float.  Numeric can't produce any more than it
was given and it was cast from float.

bray=# select 653596708775675750507850507570708696432 ::numeric;
                 numeric
-----------------------------------------
 653596708775675750507850507570708696432
(1 row)

-- 
Oliver Elphick                                          olly(at)lfix(dot)co(dot)uk
Isle of Wight                              http://www.lfix.co.uk/oliver
GPG: 1024D/A54310EA  92C8 39E7 280E 3631 3F0E  1EC0 5664 7A2F A543 10EA
                 ========================================
     "I saw in the night visions, and, behold, one like the 
      Son of man came with the clouds of heaven, and came to
      the Ancient of days, and they brought him near before 
      him. And there was given him dominion, and glory, and 
      a kingdom, that all people, nations, and languages, 
      should serve him; his dominion is an everlasting 
      dominion, which shall not pass away, and his kingdom 
      that which shall not be destroyed."     
                                    Daniel 7:13,14


In response to

Responses

pgsql-sql by date

Next:From: Jan WieckDate: 2004-08-25 00:39:22
Subject: Re: Possible rounding error of large float values?
Previous:From: Josh BerkusDate: 2004-08-24 19:52:30
Subject: Re: Possible rounding error of large float values?

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group