[PATCH 0/2] Improve ndistinct estimates for flattened UNION ALL

Hi all,

Consider a semi-join whose inner input is a UNION ALL of two large,
disjoint key sets:

SELECT *
FROM outer_relation AS o
WHERE o.id IN (
SELECT id FROM arm_one
UNION ALL
SELECT id FROM arm_two
);

PostgreSQL estimates the UNION ALL arms and the appendrel's output rows
from member relations, but it does not provide parent-column ndistinct
information to consumers outside the set operation. Once the simple
UNION ALL is pulled up, the synthetic parent has no relation OID and no
pg_statistic tuple, so most distinct-value questions about its columns
return DEFAULT_NUM_DISTINCT. A union containing millions of keys
therefore looks like a 200-distinct-value domain. That can make a
semi-join estimate that nearly every outer row matches, undercount
GROUP BY/DISTINCT output groups, or overstate Memoize hit rates.

The missing information is not a point estimate: without knowing how
member value sets overlap, PostgreSQL cannot infer the true number of
union-wide distinct values. Using one member's estimate, the old
default, or an invented midpoint would hide that uncertainty. The
planner also uses ndistinct in opposite directions: divisor-based
selectivity estimates become more selective as the count grows, while
group counts and semi-join numerators grow with it.

Proposed approach
-----------------
This series therefore derives two member-based planning ends lazily,
only when a planner consumer asks about a parent column:

* lo: the largest member's estimated group count;
* hi: the sum of all member estimates, or zero when any member is not
estimable.

The ends do not infer overlap and inherit member estimation errors, so
they are planning estimates rather than mathematical bounds. Consumers
choose the end that is conservative for the role the count plays:

* Generic distinct-value consumers normally use the count as a
divisor, so they use lo. This avoids understating equality/join
selectivity.

* GROUP BY, DISTINCT, and Memoize turn the count into an output/
cardinality estimate, so they use hi when it is complete. With
only a partial lo, they retain the used-default signal; Memoize then
remains pessimistic.

* The inner side of an equality semi/anti join uses the count as a
numerator, so it uses hi. A partial lo is accepted only after nd1
saturation; otherwise the existing uncertain-value heuristic is
retained.

The intent is to be accuracy-first: use the most accurate member-based
information available, and when overlap remains unknown, avoid
optimistic group counts, selectivity, or cache reuse. The
implementation deliberately limits itself to flattened simple UNION ALL
parents; other set operations and security-barrier views retain their
existing behavior.

Patch split
-----------
Patch 1 adds the lazy ends and the generic low-end consumer. Patch 2
adds the group/Memoize and semi-join consumer rules and their
discriminating regression tests. Splitting there keeps the common
estimation infrastructure separate from the consumer-specific safety
policy, which seems likely to be the main review discussion.

Validation
----------
Validation on x86_64 Linux, release O2 builds:

* targeted prerequisite chain: 6/6 tests passed;
* full core regression: 243/243 tests passed;
* A/B matrix: 18 low/high-cardinality scenarios, 21 runs and 2 warmups
per side, identical task/setup/query files;
* estimates were no worse than baseline in 18/18 scenarios: 9 improved
and 9 unchanged.

Median improvements included 1.817x for a high-cardinality disjoint
semi join, 1.851x for a saturated partial-lo semi join, 2.111x for a
controlled low-cardinality Memoize case, and 1.195x for a
high-cardinality anti join. No Welch-95% significant regression
remained in the final fresh-cluster run. The largest observed medians
were planning-time increases in wide/64-arm planning stress cases, but
their confidence intervals crossed zero; this is consistent with the
intentionally lazy member-estimate traversal.

Optional reviewer attachments
-----------------------------
Two optional reviewer attachments accompany this series but are not part
of the code patches:

* postgresql-flattened-union-all-ndistinct-reproducer.sql
* postgresql-flattened-union-all-ndistinct-validation-matrix.csv

The reproducer is self-contained, uses default planner settings, and
includes a natural LATERAL Top-N case: 20000 lookups over 20 keys chose
Memoize on the patched build and performed 20 index searches rather than
20000. The matrix lists all 18 low/high-cardinality scenarios, their
estimate errors, medians, Welch confidence intervals, and the controlled
Memoize settings.

Thanks,
xiangxin