Re: BUG #19524: In `contrib/btree_gist` float4/float8 GiST index operations, handling NaN values with raw C operator

Hi,

On Thu, 18 Jun 2026 at 19:32, 王跃林 <violin0613(at)tju(dot)edu(dot)cn> wrote:

> In addition to the index scan inconsistency you described, I found two
> further security-relevant effects from the same root cause.
>
> Effect 1: EXCLUDE constraint bypass (float4)
>
> CREATE TABLE reservations (
>
> room float4,
>
> during tsrange,
>
> EXCLUDE USING gist (room WITH =, during WITH &&)
>
> );
>
> INSERT INTO reservations VALUES ('NaN'::float4, '[2025-01-01,2025-01-02)');
>
> INSERT INTO reservations VALUES ('NaN'::float4, '[2025-01-01,2025-01-02)');
>
> SELECT COUNT(*) FROM reservations;
>
> Expected: 1 (second insert blocked). Actual: 2. The EXCLUDE constraint is
> silently bypassed for NaN values, allowing duplicate conflicting rows to be
> inserted.
>
> Effect 2: RLS bypass (float8)
>
> CREATE TABLE measurements (id int, val float8);
>
> CREATE INDEX ON measurements USING gist (val);
>
> INSERT INTO measurements VALUES (1, 'NaN'), (2, 1.5);
>
> ALTER TABLE measurements ENABLE ROW LEVEL SECURITY;
>
> ALTER TABLE measurements FORCE ROW LEVEL SECURITY;
>
> CREATE POLICY hide_nan ON measurements FOR SELECT USING (val !=
> 'NaN'::float8);
>
> CREATE ROLE lowpriv LOGIN;
>
> GRANT SELECT ON measurements TO lowpriv;
>
> SET ROLE lowpriv;
>
> SET enable_seqscan = off;
>
> SET enable_bitmapscan = off;
>
> SELECT * FROM measurements ORDER BY id;
>
> Expected: only (2, 1.5). Actual: both rows including (1, NaN). The RLS
> policy is bypassed when the GiST index is used because
> gbt_float8_consistent() sets *recheck = false unconditionally, so the
> heap-level RLS filter is never applied.
>
> The root cause is that gbt_float8eq(NaN, NaN) returns false due to IEEE
> 754 semantics, causing BtreeGistNotEqualStrategyNumber in
> btree_utils_num.c:300 to incorrectly conclude that NaN satisfies != NaN.
> The fix (replacing raw C operators with float8_cmp_internal) addresses both
> effects. As an additional defensive measure, *recheck in the consistent
> functions should be set to true when the query or key involves NaN.
>
I meant forwarding this on the other thread (BUG #19501)
which has been reported.

Regards,
Ayush