Re: numeric precision when raising one numeric to another.

Alvaro Herrera wrote:
> On Fri, May 20, 2005 at 01:03:08PM -0400, Tom Lane wrote:
>
> Oh, and while at it, it would be nice to solve the modulo bug that still
> lurks there:
>
> alvherre=# select 12345678901234567890 % 123;
> ?column?
> ----------
> -45
> (1 fila)
>
> alvherre=# select 12345678901234567890 % 123::numeric(4,1);
> ?column?
> ----------
> 78.0
> (1 fila)
>
> alvherre=# select 12345678901234567890 % 123::numeric(3,0);
> ?column?
> ----------
> -45
> (1 fila)
>
> alvherre=# select version();
> version
> ----------------------------------------------------------------------------------------------
> PostgreSQL 8.1devel on i686-pc-linux-gnu, compiled by GCC gcc (GCC) 3.3.6 (Debian 1:3.3.6-4)
> (1 fila)

I poked around on this one and found this in the comments in
numeric::mod_var():

/* ---------
* We do this using the equation
* mod(x,y) = x - trunc(x/y)*y
* We set rscale the same way numeric_div and numeric_mul do
* to get the right answer from the equation. The final result,
* however, need not be displayed to more precision than the inputs.
* ----------
*/

so I tried it:

test=> select 12345678901234567890 % 123;
?column?
----------
-45
(1 row)

test=> select 12345678901234567890 / 123;
?column?
--------------------
100371373180768845
(1 row)

test=> select 100371373180768845::numeric * 123::numeric;
?column?
----------------------
12345678901234567935
(1 row)

test=> select 12345678901234567890 - 12345678901234567935;
?column?
----------
-45
(1 row)

and I was quite surprised at the result. Basically, it looks like the
division is rounding _up_ the next integer on the /123 division, and
that is causing the modulo error. In fact, should the /123 round up
with numeric? I think there is an assumption in our code that div_var()
will not round up, but in fact it does in this case.

Here is 'calc' showing the same calculation:

> 12345678901234567890 % 123
78
> 12345678901234567890 / 123
~100371373180768844.63414634146341463414
> 100371373180768845 * 123

^^^^^^^^^^^^^^^^^^ rounded up by me

12345678901234567935
> 12345678901234567890 - 12345678901234567935
-45

and here is 'bc' doing integer division:

12345678901234567890 / 123
100371373180768844
100371373180768844 * 123
12345678901234567812
12345678901234567890 - 12345678901234567812
78

This is why 123::numeric(4,1) fixes it because the division returns on
digit that is truncated, rather than rounding up to the next whole
number.

I am not sure how to fix this. Adding extra scale to the division would
help, but if the division returned .999 and we had a scale of 2, it
would still round up and the truncate would not see it.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073