Gregory Stark <stark(at)enterprisedb(dot)com> writes:
> In the case reported div_var was getting 70/70 = 0.99999. Which is
> really just wrong.
Agreed, but I think your proposed patch is just a band-aid. The real
problem with div_var is that it generates inaccurate output digits at
all --- it's assuming that computing a few guard digits before rounding
will suffice to ensure that all the delivered digits are correct, but
I think there will always be corner cases where it fails.
I just blew the dust off my old copy of Knuth vol 2, and see that his
algorithm for multi-precision division generates output digits that are
correct to start with (or at least he never needs to revisit a digit
after moving on to the next). ISTM we should go over to an approach
like that. For the truncated-output case the result will be exact,
and for the rounded-output case you generate exactly one guard digit
to see if it's >= base/2.
regards, tom lane
In response to
pgsql-bugs by date
|Next:||From: Alvaro||Date: 2007-06-17 21:45:28|
|Subject: BUG #3390: initdb fails on install|
|Previous:||From: Gregory Stark||Date: 2007-06-15 16:54:46|
|Subject: Re: BUG #3387: mod on non-integer returns bad result|