Skip site navigation (1) Skip section navigation (2)

Re: BUG #3387: mod on non-integer returns bad result

From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Gregory Stark <stark(at)enterprisedb(dot)com>
Cc: "Filip Krska" <filip(dot)krska(at)comstar(dot)cz>, pgsql-bugs(at)postgresql(dot)org, "Bruce Momjian" <bruce(at)momjian(dot)us>
Subject: Re: BUG #3387: mod on non-integer returns bad result
Date: 2007-06-15 17:47:33
Message-ID: 6531.1181929653@sss.pgh.pa.us (view raw or flat)
Thread:
Lists: pgsql-bugs
Gregory Stark <stark(at)enterprisedb(dot)com> writes:
> In the case reported div_var was getting 70/70 = 0.99999. Which is
> really just wrong.

Agreed, but I think your proposed patch is just a band-aid.  The real
problem with div_var is that it generates inaccurate output digits at
all --- it's assuming that computing a few guard digits before rounding
will suffice to ensure that all the delivered digits are correct, but
I think there will always be corner cases where it fails.

I just blew the dust off my old copy of Knuth vol 2, and see that his
algorithm for multi-precision division generates output digits that are
correct to start with (or at least he never needs to revisit a digit
after moving on to the next).  ISTM we should go over to an approach
like that.  For the truncated-output case the result will be exact,
and for the rounded-output case you generate exactly one guard digit
to see if it's >= base/2.

			regards, tom lane

In response to

Responses

pgsql-bugs by date

Next:From: AlvaroDate: 2007-06-17 21:45:28
Subject: BUG #3390: initdb fails on install
Previous:From: Gregory StarkDate: 2007-06-15 16:54:46
Subject: Re: BUG #3387: mod on non-integer returns bad result

Privacy Policy | About PostgreSQL
Copyright © 1996-2014 The PostgreSQL Global Development Group