Re: [PATCH] Fix overflow and underflow in regr_r2()

From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Dean Rasheed <dean(dot)a(dot)rasheed(at)gmail(dot)com>
Cc: Chengpeng Yan <chengpeng_yan(at)outlook(dot)com>, PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: [PATCH] Fix overflow and underflow in regr_r2()
Date: 2026-05-16 16:45:16
Message-ID: 1506197.1778949916@sss.pgh.pa.us
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

BTW, on the principle of "where else did we make the same mistake",
I looked through the other aggregates using float8_regr_accum.
Most seem okay, but float8_regr_intercept does this:

PG_RETURN_FLOAT8((Sy - Sx * Sxy / Sxx) / N);

Seems to me that expression is also prone to internal
overflow/underflow. Underflow probably isn't a huge issue,
since the result will reduce to Sy/N which is likely to be good
enough. But can we do anything about overflow?

One simple change that might make things better is to compute

PG_RETURN_FLOAT8((Sy - Sx * (Sxy / Sxx)) / N);

on the theory that the sums of products are likely to both be large.

regards, tom lane

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Paul A Jungwirth 2026-05-16 16:50:35 Re: Enforce INSERT RLS checks for FOR PORTION OF leftovers?
Previous Message Hüseyin Demir 2026-05-16 16:37:31 Re: Checkpoint replication slots later