Apply extended statistics to join clause during parameterized path costing

From: Ilia Evdokimov <ilya(dot)evdokimov(at)tantorlabs(dot)com>
To: PostgreSQL Developers <pgsql-hackers(at)lists(dot)postgresql(dot)org>
Subject: Apply extended statistics to join clause during parameterized path costing
Date: 2026-08-17 11:17:08
Message-ID: 7b1bd383-fa74-4c0e-a75a-52ff9c753750@tantorlabs.com
Views: Whole Thread | Raw Message | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hi hackers,

Extended statistics are currently only applied when all clauses in a
clause list reference a single relation. When one side of an equality
clause is a Var from different relation - which is exactly what happens
while costing a parameterized inner path of a nested loop join - the
correlation between columns of the parametrized relation is silently
ignored, and the planner falls back to the standard per-column
independence assumption for those clauses.

For relations with correlated join columns this can badly estimate the
number of matching rows per probe, which in turn can lead the planner to
pick a needlessly expensive access path for the inner side.

Small example
-----------------------

```
CREATE TABLE big (a int NOT NULL, b int NOT NULL, payload text);
-- b is fully determined by a (functional dependency, degree 1.0)
INSERT INTO big SELECT i % 1000, (i % 1000) / 10, 'x' FROM
generate_series(1, 200000) i;
CREATE INDEX big_ab_idx ON big (a, b);
ANALYZE big;

CREATE TABLE small (a int NOT NULL, b int NOT NULL);
INSERT INTO small SELECT DISTINCT a, b FROM big LIMIT 50;
ANALYZE small;

-- forced to Nested Loop so we exercise the parameterized inner index scan
SET enable_mergejoin = off;
SET enable_hashjoin = off;
EXPLAIN ANALYZE SELECT * FROM big t1 JOIN small t2 ON t1.a = t2.a AND
t1.b = t2.b;
                                                           QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------
 Nested Loop  (cost=0.29..579.23 rows=100 width=18) (actual
time=0.012..2.570 rows=10000.00 loops=1)
   Buffers: shared hit=10109
   ->  Seq Scan on small t2  (cost=0.00..1.50 rows=50 width=8) (actual
time=0.004..0.006 rows=50.00 loops=1)
         Buffers: shared hit=1
   ->  Index Scan using big_ab_idx on big t1  (cost=0.29..11.53
*rows=2* width=10) (actual time=0.001..0.044 rows=200.00 loops=50)
         Index Cond: ((a = t2.a) AND (b = t2.b))
         Index Searches: 50
         Buffers: shared hit=10108
 Planning:
   Buffers: shared hit=17
 Planning Time: 0.110 ms
 Execution Time: 2.690 ms
(12 rows)

SELECT COUNT(DISTINCT a), COUNT( DISTINCT b), COUNT(DISTINCT payload)
FROM big ;
 count | count | count
-------+-------+-------
  1000 |   100 |     1
(1 row)
```

The planner estimates 2 rows per probe (independent multiplication
1/1000 * 1/100), while the actual number is 200 - a 100x underestimate
driven entirely by a -> b dependency. Now after the patch:

```
CREATE STATISTICS (dependencies) ON a, b FROM big;
ANALYZE big;
EXPLAIN ANALYZE SELECT * FROM big t1 JOIN small t2 ON t1.a = t2.a AND
t1.b = t2.b;
                                                           QUERY PLAN
---------------------------------------------------------------------------------------------------------------------------------
 Nested Loop  (cost=5.86..4474.19 rows=98 width=18) (actual
time=0.772..14.091 rows=10000.00 loops=1)
   Buffers: shared hit=9844 read=265
   ->  Seq Scan on small t2  (cost=0.00..1.50 rows=50 width=8) (actual
time=0.297..0.301 rows=50.00 loops=1)
         Buffers: shared read=1
   ->  Bitmap Heap Scan on big t1  (cost=5.86..87.45 *rows=200*
width=10) (actual time=0.040..0.253 rows=200.00 loops=50)
         Recheck Cond: ((a = t2.a) AND (b = t2.b))
         Heap Blocks: exact=10000
         Buffers: shared hit=9844 read=264
         ->  Bitmap Index Scan on big_ab_idx  (cost=0.00..5.81 rows=200
width=0) (actual time=0.014..0.014 rows=200.00 loops=50)
               Index Cond: ((a = t2.a) AND (b = t2.b))
               Index Searches: 50
               Buffers: shared hit=98 read=10
 Planning:
   Buffers: shared hit=35 read=5
 Planning Time: 0.648 ms
 Execution Time: 14.428 ms
(16 rows)
```

The row estimate (200) now matches actual exactly.

Implementation
-----------------------

A clause is now also considered compatible when it has exactly two
varnos, one of which is the 'relid' being estimated and the other
belongs to some single other relation. That other side is treated like a
pseudoconstant for the purposes of this check - its actual value doesn't
matter, only that it is fixed for the duration of one parameterized
probe. The degree logic itself is unchanged.

When clause list doesn't reference a single relation but is being
estimated for a specific varRelid - i.e. we are costing a parameterized
path - look up that relation's dependency extended statistics and apply
them the same way,via dependencies_clauselist_selectivity.

Note this only helps the `dependencies` kind of ext stats. There is
already ongoing work in this direction [0].

Any feedback are welcome.

[0]:
https://www.postgresql.org/message-id/flat/9c16f5aa-06a4-4eb0-be2e-cb7122343bc2%40tantorlabs.com#6073e26e0b3c4a03ee226fb693749396

--
Best regards,
Ilia Evdokimov,
Tantor Labs LLC,
https://tantorlabs.com/

Attachment Content-Type Size
v1-0001-Use-extended-statistics-for-join-clauses-during-p.patch text/x-patch 4.3 KB

Browse pgsql-hackers by date

  From Date Subject
Previous Message Alexander Lakhin 2026-08-17 11:00:01 Re: walsummarizer can get stuck when switching timelines