close_ps, NULLs, and DirectFunctionCall

(From IRC discussion with Andreas about some sqlsmith errors)

Commit 278148907a9 changed close_ps in geo_ops.c to return SQL NULL in
the event that a valid result point was not found (rather than crashing
or failing an assert).

But close_ps is called with DirectFunctionCall from other close_*
functions, and indirectly from dist_* functions. So while the specific
case of (point ## lseg) returns a NULL with no error in this case, for
other data types, or for certain dist_* calls, rather than a null result
you get this:

postgres=# select point(1,2) ## '((0,0),(NaN,NaN))'::box;
ERROR: function 0x9c5de0 returned NULL

postgres=# select point(1,2) <-> '((0,0),(NaN,NaN))'::box;
ERROR: function 0x9c5de0 returned NULL

This seems ... unhelpful (though it's at least better than crashing) and
inconsistent.

Does this need fixing, and if so how? The most consistent fix would seem
to be to make all the affected functions return NULL, which probably
requires factoring out a bunch of close_*_internal functions and
replacing the DirectFunctionCall usage.

--
Andrew (irc:RhodiumToad)