Re: [HACKERS] 123.45 - 123 = 0.45

From: jwieck(at)debis(dot)com (Jan Wieck) tgl(at)sss(dot)pgh(dot)pa(dot)us (Tom Lane) maillist(at)candle(dot)pha(dot)pa(dot)us, sferac(at)bo(dot)nettuno(dot)it, pgsql-hackers(at)postgreSQL(dot)org Re: [HACKERS] 123.45 - 123 = 0.45 1998-12-15 20:15:11 m0zq0sB-000EBTC@orion.SAPserv.Hamburg.dsh.de (view raw, whole thread or download thread mbox) 1998-12-15 16:43:08 from Tom Lane  1998-12-15 20:15:11 from jwieck(at)debis(dot)com (Jan Wieck)   1998-12-16 05:22:41 from Theo Kramer pgsql-hackers
```Tom Lane wrote:

> Sooner or later we ought to try to implement true fixed-point
> arbitrary-precision numeric data types per the SQL spec.  That'll be
> a lot slower than hardware float math, but its roundoff properties will
> be less surprising to novices.

I  think  the string math done in bc(1) could be a good point
to start from. In the old version (1.3 if I  remember  right)
of  the minix sources (yepp - still have them), there are the
algorithm's to get  sine,  logarithm  and  the  like  in  any
precision  as  bc  functions.  Internally, only the four base
operations, (think think) pow() and sqrt() are implemented in
C.  Anything else is done with them on the higher level.

We  need  to define what the precision of a result should be,
if it is not assigned to a column (where the precision can be
the  atttypmod).  Is  there any standard defined for? If not,

Internal representation holds different precision for DISPLAY
and CALC.

On  any operation, the DISPLAY precision is set to the higher
of the two operands.

On add/subtract, the CALC precision becomes the higher of the
two.

On multiply, the CALC precision is adjusted to hold the exact
result up to a (variable settable?) maximum.

On divide, the CALC precision is set to max and after  it  to
the number of used digits.

If  the  result get's assigned to an attribute, it is rounded
to it's atttypmod and both precisions set to that.

The types output function rounds it to the DISPLAY precision.

The  input  function  sets  both  precisions to the number of
digits present after decimal point.

Needless to say that there will be special functions to round
explicitly and set the precisions.

Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#======================================== jwieck(at)debis(dot)com (Jan Wieck) #

```

pgsql-hackers by date

 Next: From: Thomas G. Lockhart Date: 1998-12-15 20:32:20 Subject: [Fwd: catalog files] Previous: From: Thomas G. Lockhart Date: 1998-12-15 20:04:44 Subject: [Fwd: SGVLLUG DOWNLOAD: Beta of IBM DB2 5.2 for Linux]