Re: [PATCH] Fix overflow and underflow in regr_r2()

On Sat, 16 May 2026 at 22:18, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>
> Dean Rasheed <dean(dot)a(dot)rasheed(at)gmail(dot)com> writes:
> > I just had a thought: a simpler (and probably faster and more
> > accurate) solution would be to use frexp() and ldexp(), which are both
> > part of C99, so ought to be OK.
>
> Seems like a plan. We're already relying on ldexp() in pg_prng.c,
> so I doubt there's a portability issue. Reading the man page for
> frexp(), we might want to special-case Inf and NaN inputs to avoid
> assuming what it will do with those. But that would only be needed
> in the slow path where we're recovering from overflow/underflow.

OK, here's a more complete patch along those lines, intended to apply
on top of the regr_r2() patch.

I did wonder whether the final subtraction step could overflow, before
we divide by N, but I don't think it can. Except for horizontal and
vertical lines, Sxx and Syy grow quadratically compared to Sx and Sy,
so they overflow first, and it doesn't look possible for the intercept
to exceed around 1e169, other than for a horizontal line.

Regards,
Dean