BUG #19854: GiST distance ordering fails on NaN coordinates, Assert in computeDistance

From: PG Bug reporting form <noreply(at)postgresql(dot)org>
To: pgsql-bugs(at)lists(dot)postgresql(dot)org
Cc: 125atishprak(at)gmail(dot)com
Subject: BUG #19854: GiST distance ordering fails on NaN coordinates, Assert in computeDistance
Date: 2026-11-09 12:53:06
Message-ID: 19854-c255ec23cf0cb87c@postgresql.org
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-bugs

The following bug has been logged on the website:

Bug reference: 19854
Logged by: Atish Prakash
Email address: 125atishprak(at)gmail(dot)com
PostgreSQL version: 18.6
Operating system: Linux x86_64
Description:

Ordering a GiST index scan by distance fails when NaN coordinates are
involved. On an
assert-enabled build, one row with a NaN coordinate takes the server through
crash
recovery. These values are accepted by the input functions.

1. NaN in a stored box:

CREATE TABLE b(c box);
INSERT INTO b VALUES ('(NaN,0),(1,1)');
INSERT INTO b SELECT '(5,5),(6,6)' FROM generate_series(1, 5);
CREATE INDEX ON b USING gist (c);
SET enable_seqscan = off;
SET enable_sort = off;
SELECT count(*) FROM (SELECT c FROM b ORDER BY c <-> '(0,0)'::point) s;

Assert-enabled build:
TRAP: failed Assert("box->low.x <= box->high.x"), File: "gistproc.c",
Line: 1253

Without assertions (19rc1): returns 6.

Polygons and circles with a NaN coordinate crash the assert build the
same way.
Without assertions (19rc1), they fail instead with:

ERROR: inconsistent point values

2. NaN in the query point, with ordinary stored values:

CREATE TABLE p(c polygon);
INSERT INTO p SELECT '((5,5),(6,5),(6,6))' FROM generate_series(1, 5);
CREATE INDEX ON p USING gist (c);
SET enable_seqscan = off;
SET enable_sort = off;
SELECT count(*) FROM (SELECT c FROM p ORDER BY c <-> '(NaN,NaN)'::point)
s;

ERROR: index returned tuples in wrong order

This happens with and without assertions (tested with assertions on 18.6,
19rc1 and
master, and without on 19rc1). The query point is an ordinary parameter
here.

Expected: no assertion failure or internal error, whatever order NaN sorts
in.

computeDistance() in gistproc.c asserts that a box's low bound does not
exceed its high
bound, which does not hold when a bound is NaN, since its comparisons are
all false.

An AddressSanitizer build without assertions reported no memory errors in
these cases,
so this looks like a robustness problem and not a memory-safety one.

NaN handling in the geometric code has come up before, for example in
"Strange behavior
with polygon and NaN":
https://www.postgresql.org/message-id/CAGf+fX70rWFOk5cd00uMfa__0yP+vtQg5ck7c2Onb-Yczp0URA@mail.gmail.com
I did not find this assertion reported. If this is considered part of that
known
problem, please treat this as one more example of it.

I have not written a patch.

Found by comparing index and sequential scans across the operator classes
listed in the
catalog. I reproduced and checked the result myself.

Browse pgsql-bugs by date

  From Date Subject
Next Message PG Bug reporting form 2026-11-09 12:55:21 BUG #19855: GiST index on multirange misses rows for <@ with a multirange query
Previous Message Zhijie Hou 2026-11-09 11:46:59 Re: BUG #19839: Assert in ReorderBufferRestoreCleanup for aborted, spilled transaction