[PATCH 0/2] Fix two hash join costing errors in final_cost_hashjoin()

Hi,

This series fixes two independent issues in the hash join costing in
final_cost_hashjoin(), both verified on a Citus cluster (coordinator +
3 data nodes, 1000 warehouses, NVMe storage) running a mixed workload of
TPC-C-style transactions and join-heavy analytical queries.

0001: Multi-column hash keys estimate the bucket size from the largest
single-column ndistinct (the minimum over per-clause fractions), while
the output row count of the same join is estimated under the
independence assumption that calc_joinrel_size_estimate() already makes.
The same cost table therefore holds two contradictory beliefs about one
join: on a 1M x 1M three-column join with 100 distinct values per column
(one row per bucket), hash join is priced at 166,622,722 and never
chosen; with the corrected bucket size it costs 282,717 and runs in
1,946 ms versus 4,154 ms for the merge join it used to pick (-53%).
The estimate now comes from estimate_num_groups() on the clauses
estimate_multivariate_bucketsize() hands back, so bucket size and output
rows are built from the same estimator. SEMI/ANTI joins are
deliberately excluded: their probe charge is already scaled by
inner_scan_frac, and repricing it regressed a merge-semi-join query of
the workload by 27%.

0002: compute_semi_anti_join_factors() is also called for any join whose
inner side is provably unique -- plain inner joins and equally
LEFT/RIGHT/FULL joins -- but it asks clauselist_selectivity() the
semi-join question with the actual join's SpecialJoinInfo; eqjoinsel()
switches on sjinfo->jointype, so it answers with the pairwise selectivity
(1e-5) instead of the fraction of outer rows that find a match (1.0).
A 1.2M-row outer joined to a 100k unique inner then got hashjointuples =
12 in the same EXPLAIN node that prints rows = 1,200,418 -- cost and rows
contradicting each other by five orders of magnitude on one node. Fixed
by asking with a SEMI sjinfo and charging the output on the joinrel row
estimate; on the motivating query the corrected costing makes the planner
choose the merge join that measures 7% faster (986.9 ms vs 1,056.8 ms,
single shard, serial, median of 3).

Both changes affect plan choice for their shapes, so no backpatch is
planned. make check passes with the expected-file updates included; a
four-arm run of the workload with and without the series
shows no regression on the mixed workload. Three independent runs of
that harness (5 threads / 600 s, native plans, default GUCs) put both
arms inside the +/-2% run-to-run noise band every time:
+1.0% / +0.5%, -1.3% / -0.2%, and +0.9% on the second cluster.

One scope note to be clear about what the series does and does not
change: our harness plans shard tasks with parallel workers enabled,
and under parallel costing the two join types stay within 1.5% of each
other on the motivating query -- the parallel prices are accurate at
idle, and the merge join's runtime advantage there comes from cache
behavior under concurrent load, which the cost model does not observe.
Planning the same tasks serially lets 0002 choose the merge join
natively. We are not proposing any costing change for the parallel
case.

Test matrix on the workload's analytical queries: with
cpu_operator_cost reverted to its default, the default plans for two
other join-heavy queries were measurably slower than the alternatives
selected by the corrected model; the motivating query was unchanged
(single hash clause, correction does not apply).

Thanks,
He Kuang