GiST index build versus NaN coordinates

I looked into the problem reported in bug #14238,
https://www.postgresql.org/message-id/20160708151747.1426.60150@wrigleys.postgresql.org
The submitter was kind enough to give me a copy of the problem data,
and it turns out that the issue is that a few of the boxes contain
NaN coordinates. Armed with that knowledge, it's trivial to reproduce:

regression=# create table foo (f1 box);
CREATE TABLE
regression=# insert into foo values ('(3,4),(nan,5)');
INSERT 0 1
regression=# insert into foo select box(point(x,x+1),point(x,x+1)) from generate_series(1,1000) x;
INSERT 0 1000
regression=# create index on foo using gist(f1);
-- hangs, does not respond to control-C

The infinite loop is at lines 613ff in gistproc.c: once rightLower
contains a NaN, the test

while (i1 < nentries && rightLower == intervalsLower[i1].lower)

can never succeed, so i1 never increments again and the loop cannot exit.

I think that probably the most reasonable answer is to replace all the
raw "double" comparisons in this code with float8_cmp_internal() or
something that's the moral equivalent of that. Does anyone want to
propose something else?

More generally, this example makes me fearful that NaN coordinates in
geometric values are likely to cause all sorts of issues. It's too late
to disallow them, probably, but I wonder how can we identify other bugs
of this ilk.

regards, tom lane