ANALYZE-generated NaN range lengths cause 1e100-row estimates

Hi,

ANALYZE can store NaN lengths for valid range values. These statistics
can cause containment predicates on a 1020-row table to be estimated
at 1e100 rows. Query results are correct, but the estimates can lead
to poor join plans.

Reproducer, in a fresh database:

CREATE TABLE nan_len (id int PRIMARY KEY, r numrange);
INSERT INTO nan_len SELECT g, numrange(g, g + (g % 97) + 1)
FROM generate_series(1, 1000) g;
INSERT INTO nan_len SELECT 1000 + g, numrange(g, 'NaN')
FROM generate_series(1, 20) g;
ANALYZE nan_len;

EXPLAIN ANALYZE SELECT * FROM nan_len WHERE r <@ numrange(0, 200);
EXPLAIN ANALYZE SELECT * FROM nan_len WHERE r @> numrange(500, 510);

With the default statistics target, both scans estimate 1e100 rows
(printed in full by EXPLAIN), against 151 and 60 actual rows. The
length histogram contains NaN entries.

I reproduced this on assert-enabled source builds on macOS/aarch64:

18.6: REL_18_6 724edf9bde9d
19rc1: REL_19_STABLE 221423812daa
20devel: master 7787aec8ab00

Replacing the NaN-bound rows with [Infinity,Infinity] also reproduces
the problem: this is a valid, nonempty range, but its subtype
difference is NaN. The same issue affects float8-based ranges using
float8mi, and nummultirange containment estimates.

A NaN length-bin boundary reaches get_len_position(), where
interpolation produces NaN. This propagates through
calc_length_hist_frac(); the comparisons in CLAMP_PROBABILITY() do
not reject it. clamp_row_est() then converts it to 1e100. Commit
652686a334 added handling for indeterminate positions in the bounds
histogram, but get_len_position() still lacks a NaN check.

The attached patch returns 0.5 when either length-bin boundary is NaN,
in both rangetypes_selfuncs.c and multirangetypes_selfuncs.c. This
prevents NaN histogram boundaries from propagating through the
interpolation, using the same fallback as get_position(). The new
branch is not taken for histograms without NaN entries.

On all three builds, the estimates above become 168 and 47 with the
patch, using the existing statistics without another ANALYZE. In a
join test with a 2M-row table and parallel query disabled, the plan
changes from two million inner index probes into nan_len to scanning
nan_len first and making 151 probes into the larger table.

The C change passed the core regression suites on all three builds
(231/239/239 tests), using an earlier revision of the comments and
tests. The four estimate checks in the attached revision also pass
on all three.

I think this should be back-patched. The regression check using
pg_stats.range_length_histogram would need to use pg_statistic on
branches before 17.

A write-up with the reproducer, plan output and further details is at:
https://vonng.com/work/pg-range-length-nan/

Thanks,
Ruohang Feng
Vonng(at)PGSTY