pgsql: Plug holes in SP-GiST quad tree getQuadrant()

Plug holes in SP-GiST quad tree getQuadrant()

getQuadrant() places a point on each axis with the fuzzy comparisons
FPlt(), FPgt() and FPeq(), which do not cover every case. FPeq()
compares the rounded difference against EPSILON, while FPlt() and FPgt()
round the sum of one operand and EPSILON. When the centroid plus
EPSILON rounds up, a point equal to that rounded sum is more than
EPSILON above the centroid, so FPeq() is false. FPgt() compares it with
the same rounded sum, so it is false too, and FPlt() is false because
the point is above the centroid. The same gap exists below the
centroid, when the centroid minus EPSILON rounds down. No quadrant
matches, and index builds, inserts and searches fail with
"getQuadrant: impossible case".

To fix, fall back to deciding each axis separately: Use fuzzy tests
first and then tiebreak with exact comparisons. That gives every finite
value a quadrant (NaN coordinates are still an error). Points that
matched a fuzzy test before are routed as before, so existing indexes
remain valid.

Bug: #19597
Reported-by: Michael Malis <malis(at)pgrust(dot)com>
Author: Ayush Tiwari <ayushtiwari(dot)slg01(at)gmail(dot)com>
Reviewed-by: Andrey Rachitskiy <pl0h0yp1(at)gmail(dot)com>
Reviewed-by: Pierre Forstmann <pierre(dot)forstmann(at)gmail(dot)com>
Discussion: https://postgr.es/m/19597-39c532e61d78dff6@postgresql.org
Backpatch-through: 14

Branch
------
REL_17_STABLE

Details
-------
https://git.postgresql.org/pg/commitdiff/3d8c50317e01a8126eea09d1faf3786076890c64

Modified Files
--------------
src/backend/access/spgist/spgquadtreeproc.c | 43 +++++++++++++++++++++++++++--
src/test/regress/expected/spgist.out | 16 +++++++++++
src/test/regress/sql/spgist.sql | 13 +++++++++
3 files changed, 70 insertions(+), 2 deletions(-)