Re: BUG #19705: One NaN box makes a BRIN box_inclusion_ops index omit unrelated rows

Hi,

Since the cases keep turning up one by one, I ran a differential check
of v5 (0001, 0002 and the NaN point check for polygon and circle) on
master and REL_18_STABLE. For every operator pg_amop lists for the
BRIN, GiST and SP-GiST box, point, polygon and circle opclasses, with
NaN rows first, last, alone and scattered, it compares a seq scan with
the index: count(*) for searches, the first 50 distances for ordering.
6769 checks per build, built without assertions.

BRIN: 79 mismatches without the patches, none with them, on master and
on REL_18. None either with indexes built by unpatched REL_18 and
searched after the minor upgrade.

GiST on master: 376 mismatches without the patches, 91 with them. Most of what
is left is ordering: the index scan fails with "inconsistent point
values" (with assertions, Assert(box->low.y <= box->high.y) fails in
computeDistance()), with and without v5, on both branches:

CREATE TABLE b (v box);
INSERT INTO b VALUES ('(1,NaN),(0,0)');
INSERT INTO b SELECT box(point(x, y), point(x + 1, y + 1))
FROM generate_series(0, 44) x, generate_series(0, 44) y;
CREATE INDEX ON b USING gist (v);
SET enable_seqscan = off;
SELECT v <-> point '(0.5,0.5)' FROM b
ORDER BY v <-> point '(0.5,0.5)' LIMIT 3;

The seq scan returns 0, 0.5, 0.5. Circle and polygon fail the same
way with a finite query point, and all four types with a NaN in the
query point. Which LIMIT reaches the NaN entry depends on where it
lands in the tree, so v5 changes which queries fail but not that they
do. Besides that, box ordering returns NaN distances before finite
ones, and polygon ordering also gets "index returned tuples in wrong
order", with finite query points as well. The rest is point <@
polygon with a NaN in the polygon, which you already set aside.

v5-0002 does not apply to REL_18 as is: gist_box_picksplit() uses
palloc_object(), which 18 does not have. With palloc(sizeof(BOX))
there it applies, and REL_18 gives the same counts as master.

Outside this thread: SP-GiST box_ops and poly_ops show similar ordering
and ~= mismatches, unchanged by v5. quad_point_ops fails CREATE
INDEX with NaN points with "getQuadrant: impossible case", the error of
bug #19597, although that report reaches it through rounding, not NaN.
I have not checked whether the patch there covers NaN too.

The counts, the full list and the scripts are attached.

Regards,
Manu