Re: BUG #19705: One NaN box makes a BRIN box_inclusion_ops index omit unrelated rows

Hi Tomas and Heikki,

Adding you both, this touches BRIN and GiST. Here is the short version
so you don't have to read the whole thread.

One box with a NaN coordinate gets into a BRIN or GiST union key, and
no box operator is true against a NaN bound. So the whole range or
subtree holding that box disappears from index scans. KNN searches
with a NaN in a key or in the query point fail with "inconsistent
point values".

v6 from Sep 28 has four patches. 0001 fixes BRIN with a box_mergeable
support function, master only. 0002 fixes GiST union keys and the ~=
case. 0003 is Kirill's NaN point check for polygon and circle
searches, it also corrects a count in create_index.out where the index
scan has returned 4 and the seq scan 5. 0004 fixes GiST KNN. 0002 to
0004 go back to 14 with small rebases. For the back branches the BRIN
fix is v6-REL_18-0001, the scan checks whether the union contains
itself, no catalog change. That check would also do for master if you
would rather skip the catversion bump. Old indexes return correct
results without a REINDEX with either approach.

Manu compared seq scan with index scan for every operator of the box,
point, polygon and circle opclasses, 6769 checks on master. Every
mismatch goes to zero except point <@ polygon with a NaN vertex, which
is a heap bug in point_inside() and gets its own thread.

Thanks,
Shihao