From 5834a02880d13c771a14f2bdeaff02c1be17ed17 Mon Sep 17 00:00:00 2001 From: Matheus Alcantara Date: Wed, 29 Jul 2026 15:37:05 -0300 Subject: [PATCH v9 21/21] Rework hashjoin Bloom filter selection , and size filters for cheap probes The filter-pushdown code picked and costed filter candidates in a way that could build a filter per subset and treated filtering as free, and it sized the filters for accuracy rather than for cheap probing. This commit reworks both. Candidate selection and costing: * Drop the count-based candidate cap. find_interesting_bloom_filters() now returns every candidate that clears bloom_filter_pushdown_threshold instead of trimming to the bloom_filter_pushdown_max most selective ones. Selectivity, not an arbitrary count, decides what is interesting. * Cost filter-aware scan paths for real. Paths are built through the normal create_*_path constructors, and apply_expected_filters() charges one cpu_operator_cost per filter per tuple, so filtering is no longer modeled as free. A filter is chosen only when the resulting path is actually cheaper. * Apply all candidates at once instead of enumerating subsets. A single combination applies all non-overlapping candidates most-selective-first, so we no longer build one path per subset. bloom_filter_pushdown_max is repurposed to bound how many filters are applied simultaneously (the per-tuple probe cost), and the now-unused combination_floor GUC is removed. This avoids the path explosion and keeps planning time bounded. * Add an IndexPath filter variant. create_filtered_scan_path() clones the source IndexPath, reusing its indexclauses and pathkeys, and applies the same probe-cost and row adjustment as the other scan types. * Cache the interesting filters on the RelOptInfo so find_interesting_bloom_filters() runs once per rel, instead of being recomputed for core path generation and again by a CustomScan provider. * Export find_bloom_filter_combinations() and apply_expected_filters() so a CustomScan provider can build its own filter-aware paths, rather than having core construct CustomPaths on its behalf. Filter sizing: * Size hashjoin filters for cheap probes. The filters were built with bloom_create(), whose sizing favors accuracy: a 1MB bitset floor and up to MAX_HASH_FUNCS (10) hash functions. That is wrong for a filter that probes a large apply side. A filter probing tens of millions of rows then does ~10 random accesses into a bitset that does not fit in L2, so each probe is effectively a series of cache misses -- it is no longer the cheap constant-k operation the cost model assumes, and the filter can cost more than it saves even while rejecting most of the rows. Add bloom_create_probe() alongside bloom_create() (both share a static bloom_create_internal(), differing only in the bitset floor and the hash-function cap). Callers that need accuracy keep the 1MB / 10-hash sizing; the hashjoin filters in nodeHash.c use bloom_create_probe(), which uses a 1KB floor and caps the hash functions at 4. The filter is then sized to the build side and stays cache-resident, so the per-tuple probe is genuinely cheap. --- .../pg_plan_advice/expected/join_order.out | 60 ++-- .../expected/pg_stash_advice.out | 98 +++--- .../postgres_fdw/expected/postgres_fdw.out | 42 +-- src/backend/executor/nodeHash.c | 53 ++- src/backend/lib/bloomfilter.c | 16 +- src/backend/optimizer/path/allpaths.c | 317 ++++++++++-------- src/backend/optimizer/path/costsize.c | 14 +- src/backend/optimizer/path/indxpath.c | 4 +- src/backend/optimizer/path/tidpath.c | 8 +- src/backend/optimizer/plan/planner.c | 2 +- src/backend/optimizer/util/pathnode.c | 147 ++++---- src/backend/utils/misc/guc_parameters.dat | 4 +- src/backend/utils/misc/postgresql.conf.sample | 4 +- src/include/lib/bloomfilter.h | 4 + src/include/nodes/pathnodes.h | 10 + src/include/optimizer/pathnode.h | 15 +- src/include/optimizer/paths.h | 1 + .../expected/test_bloom_customscan.out | 92 ++++- .../sql/test_bloom_customscan.sql | 51 ++- .../test_bloom_customscan.c | 74 ++-- src/test/regress/expected/hashjoin_bloom.out | 8 +- .../expected/hashjoin_bloom_snowflake.out | 38 +-- .../regress/expected/hashjoin_bloom_star.out | 108 +++--- src/test/regress/expected/join.out | 74 ++-- src/test/regress/expected/misc_functions.out | 14 +- src/test/regress/expected/returning.out | 4 +- src/test/regress/expected/select_parallel.out | 37 +- 27 files changed, 779 insertions(+), 520 deletions(-) diff --git a/contrib/pg_plan_advice/expected/join_order.out b/contrib/pg_plan_advice/expected/join_order.out index ded25c4010b..a274cfa4003 100644 --- a/contrib/pg_plan_advice/expected/join_order.out +++ b/contrib/pg_plan_advice/expected/join_order.out @@ -27,29 +27,27 @@ SELECT * FROM jo_fact f LEFT JOIN jo_dim1 d1 ON f.dim1_id = d1.id LEFT JOIN jo_dim2 d2 ON f.dim2_id = d2.id WHERE val1 = 1 AND val2 = 1; - QUERY PLAN ----------------------------------------------- + QUERY PLAN +------------------------------------------------------------ Hash Join - Hash Cond: (f.dim2_id = d2.id) - -> Hash Join - Hash Cond: (f.dim1_id = d1.id) - -> Seq Scan on jo_fact f - Bloom Filter 2: keys=(dim2_id) - Bloom Filter 1: keys=(dim1_id) - -> Hash - Bloom Filter 1 + Hash Cond: ((f.dim1_id = d1.id) AND (f.dim2_id = d2.id)) + -> Seq Scan on jo_fact f + Bloom Filter 1: keys=(dim1_id, dim2_id) + -> Hash + Bloom Filter 1 + -> Nested Loop -> Seq Scan on jo_dim1 d1 Filter: (val1 = 1) - -> Hash - Bloom Filter 2 - -> Seq Scan on jo_dim2 d2 - Filter: (val2 = 1) + -> Materialize + -> Seq Scan on jo_dim2 d2 + Filter: (val2 = 1) Generated Plan Advice: - JOIN_ORDER(f d1 d2) - HASH_JOIN(d1 d2) + JOIN_ORDER(f (d1 d2)) + NESTED_LOOP_MATERIALIZE(d2) + HASH_JOIN((d1 d2)) SEQ_SCAN(f d1 d2) NO_GATHER(f d1 d2) -(20 rows) +(18 rows) -- Force a few different join orders. Some of these are very inefficient, -- but the planner considers them all viable. @@ -60,21 +58,17 @@ SELECT * FROM jo_fact f LEFT JOIN jo_dim1 d1 ON f.dim1_id = d1.id LEFT JOIN jo_dim2 d2 ON f.dim2_id = d2.id WHERE val1 = 1 AND val2 = 1; - QUERY PLAN ----------------------------------------------- + QUERY PLAN +------------------------------------------ Hash Join Hash Cond: (f.dim2_id = d2.id) -> Hash Join Hash Cond: (f.dim1_id = d1.id) -> Seq Scan on jo_fact f - Bloom Filter 2: keys=(dim2_id) - Bloom Filter 1: keys=(dim1_id) -> Hash - Bloom Filter 1 -> Seq Scan on jo_dim1 d1 Filter: (val1 = 1) -> Hash - Bloom Filter 2 -> Seq Scan on jo_dim2 d2 Filter: (val2 = 1) Supplied Plan Advice: @@ -84,7 +78,7 @@ SELECT * FROM jo_fact f HASH_JOIN(d1 d2) SEQ_SCAN(f d1 d2) NO_GATHER(f d1 d2) -(22 rows) +(18 rows) SET LOCAL pg_plan_advice.advice = 'join_order(f d2 d1)'; EXPLAIN (COSTS OFF, PLAN_ADVICE) @@ -92,21 +86,17 @@ SELECT * FROM jo_fact f LEFT JOIN jo_dim1 d1 ON f.dim1_id = d1.id LEFT JOIN jo_dim2 d2 ON f.dim2_id = d2.id WHERE val1 = 1 AND val2 = 1; - QUERY PLAN ----------------------------------------------- + QUERY PLAN +------------------------------------------ Hash Join Hash Cond: (f.dim1_id = d1.id) -> Hash Join Hash Cond: (f.dim2_id = d2.id) -> Seq Scan on jo_fact f - Bloom Filter 1: keys=(dim2_id) - Bloom Filter 2: keys=(dim1_id) -> Hash - Bloom Filter 1 -> Seq Scan on jo_dim2 d2 Filter: (val2 = 1) -> Hash - Bloom Filter 2 -> Seq Scan on jo_dim1 d1 Filter: (val1 = 1) Supplied Plan Advice: @@ -116,7 +106,7 @@ SELECT * FROM jo_fact f HASH_JOIN(d2 d1) SEQ_SCAN(f d2 d1) NO_GATHER(f d1 d2) -(22 rows) +(18 rows) SET LOCAL pg_plan_advice.advice = 'join_order(d1 f d2)'; EXPLAIN (COSTS OFF, PLAN_ADVICE) @@ -215,8 +205,8 @@ SELECT * FROM jo_fact f LEFT JOIN jo_dim1 d1 ON f.dim1_id = d1.id LEFT JOIN jo_dim2 d2 ON f.dim2_id = d2.id WHERE val1 = 1 AND val2 = 1; - QUERY PLAN ----------------------------------------------------- + QUERY PLAN +------------------------------------------------ Hash Join Hash Cond: (d2.id = f.dim2_id) -> Seq Scan on jo_dim2 d2 @@ -225,9 +215,7 @@ SELECT * FROM jo_fact f -> Hash Join Hash Cond: (f.dim1_id = d1.id) -> Seq Scan on jo_fact f - Bloom Filter 1: keys=(dim1_id) -> Hash - Bloom Filter 1 -> Seq Scan on jo_dim1 d1 Filter: (val1 = 1) Supplied Plan Advice: @@ -237,7 +225,7 @@ SELECT * FROM jo_fact f HASH_JOIN(d1 (f d1)) SEQ_SCAN(d2 f d1) NO_GATHER(f d1 d2) -(20 rows) +(18 rows) SET LOCAL pg_plan_advice.advice = 'join_order(d2 d1)'; EXPLAIN (COSTS OFF, PLAN_ADVICE) diff --git a/contrib/pg_stash_advice/expected/pg_stash_advice.out b/contrib/pg_stash_advice/expected/pg_stash_advice.out index 6372bf36c2c..f55f22349b0 100644 --- a/contrib/pg_stash_advice/expected/pg_stash_advice.out +++ b/contrib/pg_stash_advice/expected/pg_stash_advice.out @@ -57,24 +57,20 @@ EXPLAIN (COSTS OFF) SELECT * FROM aa_fact f LEFT JOIN aa_dim1 d1 ON f.dim1_id = d1.id LEFT JOIN aa_dim2 d2 ON f.dim2_id = d2.id WHERE val1 = 1 AND val2 = 1; - QUERY PLAN ----------------------------------------------- + QUERY PLAN +------------------------------------------ Hash Join - Hash Cond: (f.dim2_id = d2.id) + Hash Cond: (f.dim1_id = d1.id) -> Hash Join - Hash Cond: (f.dim1_id = d1.id) + Hash Cond: (f.dim2_id = d2.id) -> Seq Scan on aa_fact f - Bloom Filter 2: keys=(dim2_id) - Bloom Filter 1: keys=(dim1_id) -> Hash - Bloom Filter 1 - -> Seq Scan on aa_dim1 d1 - Filter: (val1 = 1) + -> Seq Scan on aa_dim2 d2 + Filter: (val2 = 1) -> Hash - Bloom Filter 2 - -> Seq Scan on aa_dim2 d2 - Filter: (val2 = 1) -(15 rows) + -> Seq Scan on aa_dim1 d1 + Filter: (val1 = 1) +(11 rows) -- Force an index scan on dim1 SELECT pg_set_stashed_advice('regress_stash', :'qid', @@ -88,26 +84,22 @@ EXPLAIN (COSTS OFF) SELECT * FROM aa_fact f LEFT JOIN aa_dim1 d1 ON f.dim1_id = d1.id LEFT JOIN aa_dim2 d2 ON f.dim2_id = d2.id WHERE val1 = 1 AND val2 = 1; - QUERY PLAN ---------------------------------------------------------------- + QUERY PLAN +--------------------------------------------------------- Hash Join - Hash Cond: (f.dim2_id = d2.id) + Hash Cond: (f.dim1_id = d1.id) -> Hash Join - Hash Cond: (f.dim1_id = d1.id) + Hash Cond: (f.dim2_id = d2.id) -> Seq Scan on aa_fact f - Bloom Filter 2: keys=(dim2_id) - Bloom Filter 1: keys=(dim1_id) -> Hash - Bloom Filter 1 - -> Index Scan using aa_dim1_pkey on aa_dim1 d1 - Filter: (val1 = 1) + -> Seq Scan on aa_dim2 d2 + Filter: (val2 = 1) -> Hash - Bloom Filter 2 - -> Seq Scan on aa_dim2 d2 - Filter: (val2 = 1) + -> Index Scan using aa_dim1_pkey on aa_dim1 d1 + Filter: (val1 = 1) Supplied Plan Advice: INDEX_SCAN(d1 aa_dim1_pkey) /* matched */ -(17 rows) +(13 rows) -- Force an alternative join order SELECT pg_set_stashed_advice('regress_stash', :'qid', @@ -121,26 +113,22 @@ EXPLAIN (COSTS OFF) SELECT * FROM aa_fact f LEFT JOIN aa_dim1 d1 ON f.dim1_id = d1.id LEFT JOIN aa_dim2 d2 ON f.dim2_id = d2.id WHERE val1 = 1 AND val2 = 1; - QUERY PLAN ----------------------------------------------- + QUERY PLAN +------------------------------------------ Hash Join Hash Cond: (f.dim2_id = d2.id) -> Hash Join Hash Cond: (f.dim1_id = d1.id) -> Seq Scan on aa_fact f - Bloom Filter 2: keys=(dim2_id) - Bloom Filter 1: keys=(dim1_id) -> Hash - Bloom Filter 1 -> Seq Scan on aa_dim1 d1 Filter: (val1 = 1) -> Hash - Bloom Filter 2 -> Seq Scan on aa_dim2 d2 Filter: (val2 = 1) Supplied Plan Advice: JOIN_ORDER(f d1 d2) /* matched */ -(17 rows) +(13 rows) -- Force an alternative join strategy SELECT pg_set_stashed_advice('regress_stash', :'qid', @@ -214,24 +202,20 @@ EXPLAIN (COSTS OFF) SELECT * FROM aa_fact f LEFT JOIN aa_dim1 d1 ON f.dim1_id = d1.id LEFT JOIN aa_dim2 d2 ON f.dim2_id = d2.id WHERE val1 = 1 AND val2 = 1; - QUERY PLAN ----------------------------------------------- + QUERY PLAN +------------------------------------------ Hash Join - Hash Cond: (f.dim2_id = d2.id) + Hash Cond: (f.dim1_id = d1.id) -> Hash Join - Hash Cond: (f.dim1_id = d1.id) + Hash Cond: (f.dim2_id = d2.id) -> Seq Scan on aa_fact f - Bloom Filter 2: keys=(dim2_id) - Bloom Filter 1: keys=(dim1_id) -> Hash - Bloom Filter 1 - -> Seq Scan on aa_dim1 d1 - Filter: (val1 = 1) + -> Seq Scan on aa_dim2 d2 + Filter: (val2 = 1) -> Hash - Bloom Filter 2 - -> Seq Scan on aa_dim2 d2 - Filter: (val2 = 1) -(15 rows) + -> Seq Scan on aa_dim1 d1 + Filter: (val1 = 1) +(11 rows) -- Test that we can list each stash individually and all of them together, -- but not a nonexistent stash. @@ -281,24 +265,20 @@ EXPLAIN (COSTS OFF) SELECT * FROM aa_fact f LEFT JOIN aa_dim1 d1 ON f.dim1_id = d1.id LEFT JOIN aa_dim2 d2 ON f.dim2_id = d2.id WHERE val1 = 1 AND val2 = 1; - QUERY PLAN ----------------------------------------------- + QUERY PLAN +------------------------------------------ Hash Join - Hash Cond: (f.dim2_id = d2.id) + Hash Cond: (f.dim1_id = d1.id) -> Hash Join - Hash Cond: (f.dim1_id = d1.id) + Hash Cond: (f.dim2_id = d2.id) -> Seq Scan on aa_fact f - Bloom Filter 2: keys=(dim2_id) - Bloom Filter 1: keys=(dim1_id) -> Hash - Bloom Filter 1 - -> Seq Scan on aa_dim1 d1 - Filter: (val1 = 1) + -> Seq Scan on aa_dim2 d2 + Filter: (val2 = 1) -> Hash - Bloom Filter 2 - -> Seq Scan on aa_dim2 d2 - Filter: (val2 = 1) -(15 rows) + -> Seq Scan on aa_dim1 d1 + Filter: (val1 = 1) +(11 rows) SELECT * FROM pg_get_advice_stashes() ORDER BY stash_name; stash_name | num_entries diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 68399224f74..ddd798790e3 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -1419,7 +1419,7 @@ SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t Foreign Scan Output: t1.c1, t2.c1, t1.c3 Relations: (public.ft1 t1) INNER JOIN (public.ft2 t2) - Remote SQL: SELECT r1."C 1", r2."C 1", r1.c3 FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r1."C 1" = r2."C 1")))) ORDER BY r1.c3 ASC NULLS LAST, r1."C 1" ASC NULLS LAST LIMIT 10::bigint OFFSET 100::bigint + Remote SQL: SELECT r1."C 1", r2."C 1", r1.c3 FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r2."C 1" = r1."C 1")))) ORDER BY r1.c3 ASC NULLS LAST, r1."C 1" ASC NULLS LAST LIMIT 10::bigint OFFSET 100::bigint (4 rows) SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t1.c1 OFFSET 100 LIMIT 10; @@ -1445,7 +1445,7 @@ SELECT t1.c1, t2.c2, t3.c3 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) JOIN ft4 t Foreign Scan Output: t1.c1, t2.c2, t3.c3, t1.c3 Relations: ((public.ft1 t1) INNER JOIN (public.ft2 t2)) INNER JOIN (public.ft4 t3) - Remote SQL: SELECT r1."C 1", r2.c2, r4.c3, r1.c3 FROM (("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r1."C 1" = r2."C 1")))) INNER JOIN "S 1"."T 3" r4 ON (((r1."C 1" = r4.c1)))) ORDER BY r1.c3 ASC NULLS LAST, r1."C 1" ASC NULLS LAST LIMIT 10::bigint OFFSET 10::bigint + Remote SQL: SELECT r1."C 1", r2.c2, r4.c3, r1.c3 FROM (("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r2."C 1" = r1."C 1")))) INNER JOIN "S 1"."T 3" r4 ON (((r1."C 1" = r4.c1)))) ORDER BY r1.c3 ASC NULLS LAST, r1."C 1" ASC NULLS LAST LIMIT 10::bigint OFFSET 10::bigint (4 rows) SELECT t1.c1, t2.c2, t3.c3 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) JOIN ft4 t3 ON (t3.c1 = t1.c1) ORDER BY t1.c3, t1.c1 OFFSET 10 LIMIT 10; @@ -2060,7 +2060,7 @@ SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t Foreign Scan Output: t1.c1, t2.c1, t1.c3, t1.*, t2.* Relations: (public.ft1 t1) INNER JOIN (public.ft2 t2) - Remote SQL: SELECT r1."C 1", r2."C 1", r1.c3, CASE WHEN (r1.*)::text IS NOT NULL THEN ROW(r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8) END, CASE WHEN (r2.*)::text IS NOT NULL THEN ROW(r2."C 1", r2.c2, r2.c3, r2.c4, r2.c5, r2.c6, r2.c7, r2.c8) END FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r1."C 1" = r2."C 1")))) ORDER BY r1.c3 ASC NULLS LAST, r1."C 1" ASC NULLS LAST LIMIT 10::bigint OFFSET 100::bigint FOR UPDATE OF r1 + Remote SQL: SELECT r1."C 1", r2."C 1", r1.c3, CASE WHEN (r1.*)::text IS NOT NULL THEN ROW(r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8) END, CASE WHEN (r2.*)::text IS NOT NULL THEN ROW(r2."C 1", r2.c2, r2.c3, r2.c4, r2.c5, r2.c6, r2.c7, r2.c8) END FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r2."C 1" = r1."C 1")))) ORDER BY r1.c3 ASC NULLS LAST, r1."C 1" ASC NULLS LAST LIMIT 10::bigint OFFSET 100::bigint FOR UPDATE OF r1 (4 rows) SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t1.c1 OFFSET 100 LIMIT 10 FOR UPDATE OF t1; @@ -2085,7 +2085,7 @@ SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t Foreign Scan Output: t1.c1, t2.c1, t1.c3, t1.*, t2.* Relations: (public.ft1 t1) INNER JOIN (public.ft2 t2) - Remote SQL: SELECT r1."C 1", r2."C 1", r1.c3, CASE WHEN (r1.*)::text IS NOT NULL THEN ROW(r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8) END, CASE WHEN (r2.*)::text IS NOT NULL THEN ROW(r2."C 1", r2.c2, r2.c3, r2.c4, r2.c5, r2.c6, r2.c7, r2.c8) END FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r1."C 1" = r2."C 1")))) ORDER BY r1.c3 ASC NULLS LAST, r1."C 1" ASC NULLS LAST LIMIT 10::bigint OFFSET 100::bigint FOR UPDATE OF r1 FOR UPDATE OF r2 + Remote SQL: SELECT r1."C 1", r2."C 1", r1.c3, CASE WHEN (r1.*)::text IS NOT NULL THEN ROW(r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8) END, CASE WHEN (r2.*)::text IS NOT NULL THEN ROW(r2."C 1", r2.c2, r2.c3, r2.c4, r2.c5, r2.c6, r2.c7, r2.c8) END FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r2."C 1" = r1."C 1")))) ORDER BY r1.c3 ASC NULLS LAST, r1."C 1" ASC NULLS LAST LIMIT 10::bigint OFFSET 100::bigint FOR UPDATE OF r1 FOR UPDATE OF r2 (4 rows) SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t1.c1 OFFSET 100 LIMIT 10 FOR UPDATE; @@ -2111,7 +2111,7 @@ SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t Foreign Scan Output: t1.c1, t2.c1, t1.c3, t1.*, t2.* Relations: (public.ft1 t1) INNER JOIN (public.ft2 t2) - Remote SQL: SELECT r1."C 1", r2."C 1", r1.c3, CASE WHEN (r1.*)::text IS NOT NULL THEN ROW(r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8) END, CASE WHEN (r2.*)::text IS NOT NULL THEN ROW(r2."C 1", r2.c2, r2.c3, r2.c4, r2.c5, r2.c6, r2.c7, r2.c8) END FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r1."C 1" = r2."C 1")))) ORDER BY r1.c3 ASC NULLS LAST, r1."C 1" ASC NULLS LAST LIMIT 10::bigint OFFSET 100::bigint FOR SHARE OF r1 + Remote SQL: SELECT r1."C 1", r2."C 1", r1.c3, CASE WHEN (r1.*)::text IS NOT NULL THEN ROW(r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8) END, CASE WHEN (r2.*)::text IS NOT NULL THEN ROW(r2."C 1", r2.c2, r2.c3, r2.c4, r2.c5, r2.c6, r2.c7, r2.c8) END FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r2."C 1" = r1."C 1")))) ORDER BY r1.c3 ASC NULLS LAST, r1."C 1" ASC NULLS LAST LIMIT 10::bigint OFFSET 100::bigint FOR SHARE OF r1 (4 rows) SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t1.c1 OFFSET 100 LIMIT 10 FOR SHARE OF t1; @@ -2136,7 +2136,7 @@ SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t Foreign Scan Output: t1.c1, t2.c1, t1.c3, t1.*, t2.* Relations: (public.ft1 t1) INNER JOIN (public.ft2 t2) - Remote SQL: SELECT r1."C 1", r2."C 1", r1.c3, CASE WHEN (r1.*)::text IS NOT NULL THEN ROW(r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8) END, CASE WHEN (r2.*)::text IS NOT NULL THEN ROW(r2."C 1", r2.c2, r2.c3, r2.c4, r2.c5, r2.c6, r2.c7, r2.c8) END FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r1."C 1" = r2."C 1")))) ORDER BY r1.c3 ASC NULLS LAST, r1."C 1" ASC NULLS LAST LIMIT 10::bigint OFFSET 100::bigint FOR SHARE OF r1 FOR SHARE OF r2 + Remote SQL: SELECT r1."C 1", r2."C 1", r1.c3, CASE WHEN (r1.*)::text IS NOT NULL THEN ROW(r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8) END, CASE WHEN (r2.*)::text IS NOT NULL THEN ROW(r2."C 1", r2.c2, r2.c3, r2.c4, r2.c5, r2.c6, r2.c7, r2.c8) END FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r2."C 1" = r1."C 1")))) ORDER BY r1.c3 ASC NULLS LAST, r1."C 1" ASC NULLS LAST LIMIT 10::bigint OFFSET 100::bigint FOR SHARE OF r1 FOR SHARE OF r2 (4 rows) SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t1.c1 OFFSET 100 LIMIT 10 FOR SHARE; @@ -2165,7 +2165,7 @@ WITH t (c1_1, c1_3, c2_1) AS MATERIALIZED (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t -> Foreign Scan Output: t1.c1, t1.c3, t2.c1 Relations: (public.ft1 t1) INNER JOIN (public.ft2 t2) - Remote SQL: SELECT r1."C 1", r1.c3, r2."C 1" FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r1."C 1" = r2."C 1")))) + Remote SQL: SELECT r1."C 1", r1.c3, r2."C 1" FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r2."C 1" = r1."C 1")))) -> Sort Output: t.c1_1, t.c2_1, t.c1_3 Sort Key: t.c1_3, t.c1_1 @@ -2196,7 +2196,7 @@ SELECT t1.ctid, t1, t2, t1.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER B Foreign Scan Output: t1.ctid, t1.*, t2.*, t1.c1, t1.c3 Relations: (public.ft1 t1) INNER JOIN (public.ft2 t2) - Remote SQL: SELECT r1.ctid, CASE WHEN (r1.*)::text IS NOT NULL THEN ROW(r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8) END, CASE WHEN (r2.*)::text IS NOT NULL THEN ROW(r2."C 1", r2.c2, r2.c3, r2.c4, r2.c5, r2.c6, r2.c7, r2.c8) END, r1."C 1", r1.c3 FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r1."C 1" = r2."C 1")))) ORDER BY r1.c3 ASC NULLS LAST, r1."C 1" ASC NULLS LAST LIMIT 10::bigint OFFSET 100::bigint + Remote SQL: SELECT r1.ctid, CASE WHEN (r1.*)::text IS NOT NULL THEN ROW(r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8) END, CASE WHEN (r2.*)::text IS NOT NULL THEN ROW(r2."C 1", r2.c2, r2.c3, r2.c4, r2.c5, r2.c6, r2.c7, r2.c8) END, r1."C 1", r1.c3 FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r2."C 1" = r1."C 1")))) ORDER BY r1.c3 ASC NULLS LAST, r1."C 1" ASC NULLS LAST LIMIT 10::bigint OFFSET 100::bigint (4 rows) -- SEMI JOIN @@ -2207,7 +2207,7 @@ SELECT t1.c1 FROM ft1 t1 WHERE EXISTS (SELECT 1 FROM ft2 t2 WHERE t1.c1 = t2.c1) Foreign Scan Output: t1.c1 Relations: (public.ft1 t1) SEMI JOIN (public.ft2 t2) - Remote SQL: SELECT r1."C 1" FROM "S 1"."T 1" r1 WHERE EXISTS (SELECT NULL FROM "S 1"."T 1" r2 WHERE ((r1."C 1" = r2."C 1"))) ORDER BY r1."C 1" ASC NULLS LAST LIMIT 10::bigint OFFSET 100::bigint + Remote SQL: SELECT r1."C 1" FROM "S 1"."T 1" r1 WHERE EXISTS (SELECT NULL FROM "S 1"."T 1" r2 WHERE ((r2."C 1" = r1."C 1"))) ORDER BY r1."C 1" ASC NULLS LAST LIMIT 10::bigint OFFSET 100::bigint (4 rows) SELECT t1.c1 FROM ft1 t1 WHERE EXISTS (SELECT 1 FROM ft2 t2 WHERE t1.c1 = t2.c1) ORDER BY t1.c1 OFFSET 100 LIMIT 10; @@ -2408,7 +2408,7 @@ SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) WHERE t1.c8 = t2. Output: t1.c1, t2.c1, t1.c3 Filter: (t1.c8 = t2.c8) Relations: (public.ft1 t1) INNER JOIN (public.ft2 t2) - Remote SQL: SELECT r1."C 1", r2."C 1", r1.c3, r1.c8, r2.c8 FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r1."C 1" = r2."C 1")))) + Remote SQL: SELECT r1."C 1", r2."C 1", r1.c3, r1.c8, r2.c8 FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r2."C 1" = r1."C 1")))) (10 rows) SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) WHERE t1.c8 = t2.c8 ORDER BY t1.c3, t1.c1 OFFSET 100 LIMIT 10; @@ -2446,11 +2446,11 @@ SELECT t1c1, avg(t1c1 + t2c1) FROM (SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 -> Foreign Scan Output: t1.c1, t2.c1 Relations: (public.ft1 t1) INNER JOIN (public.ft2 t2) - Remote SQL: SELECT r1."C 1", r2."C 1" FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r1."C 1" = r2."C 1")))) + Remote SQL: SELECT r1."C 1", r2."C 1" FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r2."C 1" = r1."C 1")))) -> Foreign Scan Output: t1_1.c1, t2_1.c1 Relations: (public.ft1 t1_1) INNER JOIN (public.ft2 t2_1) - Remote SQL: SELECT r1."C 1", r2."C 1" FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r1."C 1" = r2."C 1")))) + Remote SQL: SELECT r1."C 1", r2."C 1" FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r2."C 1" = r1."C 1")))) (20 rows) SELECT t1c1, avg(t1c1 + t2c1) FROM (SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) UNION SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) AS t (t1c1, t2c1) GROUP BY t1c1 ORDER BY t1c1 OFFSET 100 LIMIT 10; @@ -2489,7 +2489,7 @@ SELECT t1."C 1" FROM "S 1"."T 1" t1, LATERAL (SELECT DISTINCT t2.c1, t3.c1 FROM -> Foreign Scan Output: t2.c1, t3.c1 Relations: (public.ft1 t2) INNER JOIN (public.ft2 t3) - Remote SQL: SELECT r1."C 1", r2."C 1" FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r1."C 1" = r2."C 1")) AND ((r1.c2 = $1::integer)))) + Remote SQL: SELECT r1."C 1", r2."C 1" FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r2."C 1" = r1."C 1")) AND ((r1.c2 = $1::integer)))) (17 rows) SELECT t1."C 1" FROM "S 1"."T 1" t1, LATERAL (SELECT DISTINCT t2.c1, t3.c1 FROM ft1 t2, ft2 t3 WHERE t2.c1 = t3.c1 AND t2.c2 = t1.c2) q ORDER BY t1."C 1" OFFSET 10 LIMIT 10; @@ -2520,7 +2520,7 @@ SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1 AND CURRENT_USER = -> Foreign Scan Output: t1.c1, t1.c3, t2.c1 Relations: (public.ft1 t1) INNER JOIN (public.ft2 t2) - Remote SQL: SELECT r1."C 1", r2."C 1", r1.c3 FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r1."C 1" = r2."C 1")))) ORDER BY r1.c3 ASC NULLS LAST, r1."C 1" ASC NULLS LAST + Remote SQL: SELECT r1."C 1", r2."C 1", r1.c3 FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r2."C 1" = r1."C 1")))) ORDER BY r1.c3 ASC NULLS LAST, r1."C 1" ASC NULLS LAST (9 rows) -- non-Var items in targetlist of the nullable rel of a join preventing @@ -2623,7 +2623,7 @@ SELECT * FROM ft1, ft2, ft4, ft5, local_tbl WHERE ft1.c1 = ft2.c1 AND ft1.c2 = f -> Foreign Scan Output: ft1.c1, ft1.c2, ft1.c3, ft1.c4, ft1.c5, ft1.c6, ft1.c7, ft1.c8, ft1.*, ft2.c1, ft2.c2, ft2.c3, ft2.c4, ft2.c5, ft2.c6, ft2.c7, ft2.c8, ft2.*, ft4.c1, ft4.c2, ft4.c3, ft4.*, ft5.c1, ft5.c2, ft5.c3, ft5.* Relations: (((public.ft1) INNER JOIN (public.ft2)) INNER JOIN (public.ft4)) INNER JOIN (public.ft5) - Remote SQL: SELECT r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8, CASE WHEN (r1.*)::text IS NOT NULL THEN ROW(r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8) END, r2."C 1", r2.c2, r2.c3, r2.c4, r2.c5, r2.c6, r2.c7, r2.c8, CASE WHEN (r2.*)::text IS NOT NULL THEN ROW(r2."C 1", r2.c2, r2.c3, r2.c4, r2.c5, r2.c6, r2.c7, r2.c8) END, r3.c1, r3.c2, r3.c3, CASE WHEN (r3.*)::text IS NOT NULL THEN ROW(r3.c1, r3.c2, r3.c3) END, r4.c1, r4.c2, r4.c3, CASE WHEN (r4.*)::text IS NOT NULL THEN ROW(r4.c1, r4.c2, r4.c3) END FROM ((("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r1."C 1" = r2."C 1")) AND ((r2."C 1" < 100)) AND ((r1."C 1" < 100)))) INNER JOIN "S 1"."T 3" r3 ON (((r1.c2 = r3.c1)))) INNER JOIN "S 1"."T 4" r4 ON (((r1.c2 = r4.c1)))) ORDER BY r1.c2 ASC NULLS LAST FOR UPDATE OF r1 FOR UPDATE OF r2 FOR UPDATE OF r3 FOR UPDATE OF r4 + Remote SQL: SELECT r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8, CASE WHEN (r1.*)::text IS NOT NULL THEN ROW(r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8) END, r2."C 1", r2.c2, r2.c3, r2.c4, r2.c5, r2.c6, r2.c7, r2.c8, CASE WHEN (r2.*)::text IS NOT NULL THEN ROW(r2."C 1", r2.c2, r2.c3, r2.c4, r2.c5, r2.c6, r2.c7, r2.c8) END, r3.c1, r3.c2, r3.c3, CASE WHEN (r3.*)::text IS NOT NULL THEN ROW(r3.c1, r3.c2, r3.c3) END, r4.c1, r4.c2, r4.c3, CASE WHEN (r4.*)::text IS NOT NULL THEN ROW(r4.c1, r4.c2, r4.c3) END FROM ((("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r2."C 1" = r1."C 1")) AND ((r2."C 1" < 100)) AND ((r1."C 1" < 100)))) INNER JOIN "S 1"."T 3" r3 ON (((r1.c2 = r3.c1)))) INNER JOIN "S 1"."T 4" r4 ON (((r1.c2 = r4.c1)))) ORDER BY r1.c2 ASC NULLS LAST FOR UPDATE OF r1 FOR UPDATE OF r2 FOR UPDATE OF r3 FOR UPDATE OF r4 -> Merge Join Output: ft1.c1, ft1.c2, ft1.c3, ft1.c4, ft1.c5, ft1.c6, ft1.c7, ft1.c8, ft1.*, ft2.c1, ft2.c2, ft2.c3, ft2.c4, ft2.c5, ft2.c6, ft2.c7, ft2.c8, ft2.*, ft4.c1, ft4.c2, ft4.c3, ft4.*, ft5.c1, ft5.c2, ft5.c3, ft5.* Merge Cond: (ft1.c2 = ft5.c1) @@ -3085,7 +3085,7 @@ select sum(t1.c1), count(t2.c1) from ft1 t1 inner join ft2 t2 on (t1.c1 = t2.c1) Output: t1.c1 Filter: (((((t1.c1 * t2.c1) / (t1.c1 * t2.c1)))::double precision * random()) <= '1'::double precision) Relations: (public.ft1 t1) INNER JOIN (public.ft2 t2) - Remote SQL: SELECT r1."C 1", r2."C 1" FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r1."C 1" = r2."C 1")))) + Remote SQL: SELECT r1."C 1", r2."C 1" FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r2."C 1" = r1."C 1")))) (7 rows) -- GROUP BY clause having expressions @@ -4295,7 +4295,7 @@ EXPLAIN (VERBOSE, COSTS OFF) EXECUTE st2(10, 20); ---------------------------------------------------------------------------------------------------------------------------------- Nested Loop Semi Join Output: t1.c1, t1.c2, t1.c3, t1.c4, t1.c5, t1.c6, t1.c7, t1.c8 - Join Filter: (t1.c3 = t2.c3) + Join Filter: (t2.c3 = t1.c3) -> Foreign Scan on public.ft1 t1 Output: t1.c1, t1.c2, t1.c3, t1.c4, t1.c5, t1.c6, t1.c7, t1.c8 Remote SQL: SELECT "C 1", c2, c3, c4, c5, c6, c7, c8 FROM "S 1"."T 1" WHERE (("C 1" < 20)) ORDER BY "C 1" ASC NULLS LAST @@ -4328,7 +4328,7 @@ EXPLAIN (VERBOSE, COSTS OFF) EXECUTE st3(10, 20); Foreign Scan Output: t1.c1, t1.c2, t1.c3, t1.c4, t1.c5, t1.c6, t1.c7, t1.c8 Relations: (public.ft1 t1) SEMI JOIN (public.ft2 t2) - Remote SQL: SELECT r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8 FROM "S 1"."T 1" r1 WHERE ((r1."C 1" < 20)) AND EXISTS (SELECT NULL FROM "S 1"."T 1" r3 WHERE ((r3."C 1" > 10)) AND ((date(r3.c5) = '1970-01-17'::date)) AND ((r1.c3 = r3.c3))) ORDER BY r1."C 1" ASC NULLS LAST + Remote SQL: SELECT r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8 FROM "S 1"."T 1" r1 WHERE ((r1."C 1" < 20)) AND EXISTS (SELECT NULL FROM "S 1"."T 1" r3 WHERE ((r3."C 1" > 10)) AND ((date(r3.c5) = '1970-01-17'::date)) AND ((r3.c3 = r1.c3))) ORDER BY r1."C 1" ASC NULLS LAST (4 rows) EXECUTE st3(10, 20); @@ -5113,7 +5113,7 @@ EXPLAIN (verbose, costs off) Foreign Scan Output: ft2.c1, ft2.c2, ft2.c3, ft2.c4, ft2.c5, ft2.c6, ft2.c7, ft2.c8, ft4.c1, ft4.c2, ft4.c3 Relations: ((((public.ft2) INNER JOIN (public.ft4)) SEMI JOIN (public.ft2 ft2_1)) INNER JOIN (public.ft2 ft2_2)) SEMI JOIN (public.ft4 ft4_1) - Remote SQL: SELECT r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8, r6.c1, r6.c2, r6.c3 FROM (("S 1"."T 1" r1 INNER JOIN "S 1"."T 3" r6 ON (((r1.c2 = r6.c1)) AND ((r1."C 1" > 900)))) INNER JOIN "S 1"."T 1" r8 ON (((r1.c2 = r8.c2)))) WHERE EXISTS (SELECT NULL FROM "S 1"."T 3" r9 WHERE ((r1.c2 = r9.c2))) AND EXISTS (SELECT NULL FROM "S 1"."T 1" r7 WHERE ((r6.c2 = r7.c2))) ORDER BY r1."C 1" ASC NULLS LAST LIMIT 10::bigint + Remote SQL: SELECT r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8, r6.c1, r6.c2, r6.c3 FROM (("S 1"."T 1" r1 INNER JOIN "S 1"."T 3" r6 ON (((r1.c2 = r6.c1)) AND ((r1."C 1" > 900)))) INNER JOIN "S 1"."T 1" r8 ON (((r1.c2 = r8.c2)))) WHERE EXISTS (SELECT NULL FROM "S 1"."T 3" r9 WHERE ((r1.c2 = r9.c2))) AND EXISTS (SELECT NULL FROM "S 1"."T 1" r7 WHERE ((r7.c2 = r6.c2))) ORDER BY r1."C 1" ASC NULLS LAST LIMIT 10::bigint (4 rows) SELECT ft2.*, ft4.* FROM ft2 INNER JOIN @@ -5184,7 +5184,7 @@ SELECT ft1.c1 FROM ft1 JOIN ft2 on ft1.c1 = ft2.c1 WHERE -> Foreign Scan Output: ft1.c1, ft2.c1 Relations: (public.ft1) INNER JOIN (public.ft2) - Remote SQL: SELECT r1."C 1", r2."C 1" FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r1."C 1" = r2."C 1")))) ORDER BY r1."C 1" ASC NULLS LAST + Remote SQL: SELECT r1."C 1", r2."C 1" FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r2."C 1" = r1."C 1")))) ORDER BY r1."C 1" ASC NULLS LAST -> Foreign Scan Output: ft2_1.c1, ft4.c1 Relations: (public.ft2 ft2_1) INNER JOIN (public.ft4) @@ -7287,7 +7287,7 @@ UPDATE remt2 SET c2 = remt2.c2 || remt2.c2 FROM loct1 WHERE loct1.c1 = remt2.c1 Remote SQL: UPDATE public.loct2 SET c2 = $2 WHERE ctid = $1 RETURNING c1, c2 -> Nested Loop Output: (remt2.c2 || remt2.c2), remt2.ctid, remt2.*, loct1.ctid - Join Filter: (remt2.c1 = loct1.c1) + Join Filter: (loct1.c1 = remt2.c1) -> Seq Scan on public.loct1 Output: loct1.ctid, loct1.c1 -> Foreign Scan on public.remt2 diff --git a/src/backend/executor/nodeHash.c b/src/backend/executor/nodeHash.c index 56d15890a9c..41a6a3431bb 100644 --- a/src/backend/executor/nodeHash.c +++ b/src/backend/executor/nodeHash.c @@ -82,6 +82,17 @@ static bool ExecParallelHashTuplePrealloc(HashJoinTable hashtable, static void ExecParallelHashMergeCounters(HashJoinTable hashtable); static void ExecParallelHashCloseBatchAccessors(HashJoinTable hashtable); +static bloom_filter *bloom_create_probe(int64 total_elems, int bloom_work_mem, uint64 seed); + + +/* + * Sizing for probe-oriented Bloom filters. A small bitset floor, so small + * build sides stay resident in CPU cache, and a low cap on the number of hash + * functions, since each is an independent (potentially cache-missing) bitset + * access on every probe. + */ +#define BLOOM_PROBE_MIN_BITSET_BYTES 1024 +#define BLOOM_PROBE_MAX_HASH_FUNCS 4 /* ---------------------------------------------------------------- * ExecHash @@ -714,8 +725,10 @@ ExecHashTableCreate(HashState *state) * same lifecycle (during rescans, etc.). * * The size of the filter is bounded by both the estimated inner row count - * and a fixed fraction of work_mem. bloom_create() will round down to - * the next power-of-two bitset and enforces a 1MB minimum. + * and a fixed fraction of work_mem. We use bloom_create_probe(), which + * (unlike bloom_create) sizes the bitset to the build side without a 1MB + * floor and caps the hash-function count, so the filter stays small + * enough to be cache-resident and cheap to probe per tuple. * * XXX This may need more thought. If we limit bloom_work_mem too much, * the false positive rate will get too bad, and we won't filter enough @@ -739,8 +752,8 @@ ExecHashTableCreate(HashState *state) bloom_work_mem = Max(1024, work_mem / 8); oldctx = MemoryContextSwitchTo(hashtable->hashCxt); - state->bloom_filter = bloom_create((int64) Max(rows, 1.0), - bloom_work_mem, 0); + state->bloom_filter = bloom_create_probe((int64) Max(rows, 1.0), + bloom_work_mem, 0); /* * If a recipient opted in, also build one filter per join key (in @@ -753,8 +766,8 @@ ExecHashTableCreate(HashState *state) state->perkey_filters = palloc_array(struct bloom_filter *, state->perkey_nfilters); for (int i = 0; i < state->perkey_nfilters; i++) - state->perkey_filters[i] = bloom_create((int64) Max(rows, 1.0), - bloom_work_mem, 0); + state->perkey_filters[i] = bloom_create_probe((int64) Max(rows, 1.0), + bloom_work_mem, 0); } MemoryContextSwitchTo(oldctx); @@ -3784,3 +3797,31 @@ get_hash_memory_limit(void) return (size_t) mem_limit; } + +/* + * Create a Bloom filter tuned for repeated per-tuple probing within a query, + * rather than one built once and checked in bulk. + * + * bloom_create's sizing is a poor fit for that use in two ways, both of which + * make each probe expensive: + * + * - Size. bloom_create enforces a 1MB minimum bitset, which for a typical + * build side does not fit in the CPU's L2 cache, so every probe of a large + * scan incurs cache misses. Here we drop that floor and let the bitset be + * sized to the build side, so small build sides yield cache-resident filters. + * + * - Hash functions. Each of the k hash functions is an independent random + * access into the bitset, so a high k means many (cache-missing) accesses per + * probe. We cap k well below MAX_HASH_FUNCS, trading a slightly higher + * false-positive rate for much cheaper probes. + * + * This mirrors the "small fixed number of hash functions" and "keep the filter + * within the L2 cache" choices in the bottom-up Bloom filter paper. + */ +bloom_filter * +bloom_create_probe(int64 total_elems, int bloom_work_mem, uint64 seed) +{ + return bloom_create_custom(total_elems, bloom_work_mem, + BLOOM_PROBE_MIN_BITSET_BYTES, + BLOOM_PROBE_MAX_HASH_FUNCS, seed); +} diff --git a/src/backend/lib/bloomfilter.c b/src/backend/lib/bloomfilter.c index fbcf788e271..9f2f9006ab7 100644 --- a/src/backend/lib/bloomfilter.c +++ b/src/backend/lib/bloomfilter.c @@ -84,7 +84,8 @@ static inline uint32 mod_m(uint32 val, uint64 m); * 0. Callers can also use a pseudo-random seed, eg from pg_prng_uint64(). */ bloom_filter * -bloom_create(int64 total_elems, int bloom_work_mem, uint64 seed) +bloom_create_custom(int64 total_elems, int bloom_work_mem, + uint64 min_bitset_bytes, int max_hash_funcs, uint64 seed) { bloom_filter *filter; int bloom_power; @@ -99,7 +100,7 @@ bloom_create(int64 total_elems, int bloom_work_mem, uint64 seed) * false positive rate still won't exceed 2% in almost all cases. */ bitset_bytes = Min(bloom_work_mem * UINT64CONST(1024), total_elems * 2); - bitset_bytes = Max(1024 * 1024, bitset_bytes); + bitset_bytes = Max(min_bitset_bytes, bitset_bytes); /* * Size in bits should be the highest power of two <= target. bitset_bits @@ -112,13 +113,22 @@ bloom_create(int64 total_elems, int bloom_work_mem, uint64 seed) /* Allocate bloom filter with unset bitset */ filter = palloc0(offsetof(bloom_filter, bitset) + sizeof(unsigned char) * bitset_bytes); - filter->k_hash_funcs = optimal_k(bitset_bits, total_elems); + filter->k_hash_funcs = Min(optimal_k(bitset_bits, total_elems), + max_hash_funcs); filter->seed = seed; filter->m = bitset_bits; return filter; } +bloom_filter * +bloom_create(int64 total_elems, int bloom_work_mem, uint64 seed) +{ + return bloom_create_custom(total_elems, bloom_work_mem, + UINT64CONST(1024) * 1024, MAX_HASH_FUNCS, + seed); +} + /* * Free Bloom filter */ diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 62000d9fe58..4bef76c086a 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -877,7 +877,7 @@ set_plain_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte) return; /* Consider sequential scan */ - add_path(rel, create_seqscan_path(root, rel, required_outer, 0)); + add_path(rel, create_seqscan_path(root, rel, required_outer, 0, NIL)); /* If appropriate, consider parallel sequential scan */ if (rel->consider_parallel && required_outer == NULL) @@ -904,7 +904,7 @@ create_plain_partial_paths(PlannerInfo *root, RelOptInfo *rel) return; /* Add an unordered partial path based on a parallel sequential scan. */ - add_partial_path(rel, create_seqscan_path(root, rel, NULL, parallel_workers)); + add_partial_path(rel, create_seqscan_path(root, rel, NULL, parallel_workers, NIL)); } /* @@ -1096,9 +1096,10 @@ bloom_build_side_join_ratio(PlannerInfo *root, List *build_sides, * fraction is estimated as the semijoin selectivity of those clauses. * * A candidate is "interesting" only if it is expected to eliminate at least - * bloom_filter_pushdown_threshold of the rel's tuples. We keep at most - * bloom_filter_pushdown_max of the most selective candidates, and return them - * as a list of ExpectedFilter nodes. + * bloom_filter_pushdown_threshold of the rel's tuples. Every candidate that + * clears that bar is returned as an ExpectedFilter node; deciding which of + * them to actually combine into a scan path (and how many) is left to + * find_bloom_filter_combinations(). * * XXX This needs to be careful to not interfere with the general selectivity * estimation, performed by clauselist_selectivity(). We'll estimate the filter @@ -1128,6 +1129,15 @@ find_interesting_bloom_filters(PlannerInfo *root, RelOptInfo *rel) if (rel->reloptkind != RELOPT_BASEREL) return NIL; + /* + * Return the cached result if we've already computed the interesting + * filters for this rel. Both core path generation and a CustomScan + * provider may ask for them, and the enumeration/selectivity work below + * is not free, so we do it only once per base relation. + */ + if (rel->bloom_filters_valid) + return rel->bloom_filters; + /* Collect candidate hashjoinable equality clauses for this rel. */ candidates = generate_implied_equalities_for_all_columns(root, rel, bloom_em_matches_anybarevar, @@ -1419,34 +1429,13 @@ find_interesting_bloom_filters(PlannerInfo *root, RelOptInfo *rel) } /* - * We have collected all potentially intresting filters. Evaluate - * selectivity of each group and keep only the most interesting filters. - * Filters have to eliminate at least bloom_filter_pushdown_threshold - * tuples, and we keep only bloom_filter_pushdown_max most selective ones. - * - * We only connsider a limited number of interesting filters, to prevent - * path explosion. If we found too many, keep only the most selective ones - * (with smallest surviving fraction of tuples), to bound the number of - * generated paths. - * - * XXX This also aligns with good join orders - those tend to perform the - * most selective joins first. So we get to build the filters soon, even - * if the hashjoin optimization is not disabled. + * Return every candidate that clears bloom_filter_pushdown_threshold. + * Which of them to combine into a scan path, and how many, is decided by + * find_bloom_filter_combinations(); a filter that is weak on its own can + * still be useful combined with others, or to a smart consumer. */ - while (list_length(result) > bloom_filter_pushdown_max) - { - ExpectedFilter *worst = NULL; - ListCell *lcw; - - foreach(lcw, result) - { - ExpectedFilter *f = (ExpectedFilter *) lfirst(lcw); - - if (worst == NULL || f->selectivity > worst->selectivity) - worst = f; - } - result = list_delete_ptr(result, worst); - } + rel->bloom_filters = result; + rel->bloom_filters_valid = true; return result; } @@ -1456,59 +1445,40 @@ find_interesting_bloom_filters(PlannerInfo *root, RelOptInfo *rel) * Generate additional scan paths that anticipate one or more pushed-down * Bloom filters. * - * For each non-empty subset of the interesting filters, we clone every eligible - * existing scan path, reducing its row estimate by the combined selectivity and - * attaching the corresponding ExpectedFilter nodes. + * For each combination of the interesting filters (see + * find_bloom_filter_combinations), we build a new path via the real + * constructor for that path's type, passing the filters in so the + * constructor's apply_expected_filters() call charges a per-tuple probe cost. + * + * IndexPath is the one exception: it goes through create_filtered_scan_path, + * which clones the source path, because create_index_path() would re-derive + * its indexclauses/pathkeys from scratch for no benefit (see the comment on + * create_filtered_scan_path). * * These paths are kept alongside the regular paths (add_path keeps paths with * differing expected_filters) and are consumed by join path generation; * set_cheapest never selects them. * - * XXX We must not clone paths that already have expected filters. - * - * XXX The cloning is a rather dirty way to copy paths. It does not readjust the - * cost in a reasonable way. For example custom scans could do something smart - * with the filters, so it should have a chance to deal with that. A cleaner - * solution might be to actually pass the filters to the various "create" - * function, like create_seqscan_path/... For CustomScan nodes we can probably - * do most of this in the set_rel_pathlist_hook, somewhere. Maybe that needs - * some helper methods, though. And maybe it will need to pass some of the info - * through the callbacks? Not sure, someone has to try that. - * - * XXX This may need some major changes to work with custom scans. Right now we - * only consider filters exactly matching the hash keys, so if the hashjoin is - * on (t1.a = t2.a AND t1.b = t2.b), then the filter will be on (a,b). But a - * custom scan may prefer "split" filters on each column independently. We'd - * need a way for the custom scan to indicate that, and we'd need to apply this - * only to the "matching" scan paths (and not to any other scan paths). But - * we only look at the paths after selecting the "interesting" filters, so we'd - * need to rethink that - we'd need to make the "interesting" filters specific - * to a path, or something like that. + * CustomPath is not handled here at all. Core cannot safely copy or construct + * a CustomPath, so a provider that wants filter-aware CustomPaths must build + * them itself, inside its own set_rel_pathlist_hook, by calling + * find_bloom_filter_combinations() directly and costing the result however it + * likes -- with real smart costing, or by calling apply_expected_filters() on + * its own freshly-built CustomPath if it doesn't want to be smart. */ static void generate_expected_filter_paths(PlannerInfo *root, RelOptInfo *rel) { - List *filters; + List *combinations; List *basepaths = NIL; - int nfilters; - uint32 combo; - ListCell *lc; - - filters = find_interesting_bloom_filters(root, rel); - if (filters == NIL) - return; - - nfilters = list_length(filters); /* * Snapshot the existing unparameterized, non-partial scan paths of a * supported type. We must snapshot before calling add_path(), which * mutates rel->pathlist. */ - foreach(lc, rel->pathlist) + foreach_ptr(Path, path, rel->pathlist) { - Path *path = (Path *) lfirst(lc); - /* XXX Is parameterization really a problem? Always? */ if (path->param_info != NULL || path->expected_filters != NIL) continue; @@ -1522,19 +1492,6 @@ generate_expected_filter_paths(PlannerInfo *root, RelOptInfo *rel) case T_TidRangePath: basepaths = lappend(basepaths, path); break; - case T_CustomPath: - - /* - * A base-relation CustomScan can receive a pushed-down - * filter, but only if the provider advertised that it knows - * how to consume one (it probes the filter itself inside its - * scan loop; see find_bloom_filter_recipient in createplan.c - * and ExecInitCustomScan). Providers that don't opt in, like - * heap, are unaffected. - */ - if (((CustomPath *) path)->flags & CUSTOMPATH_SUPPORT_BLOOM_FILTERS) - basepaths = lappend(basepaths, path); - break; default: break; } @@ -1543,79 +1500,155 @@ generate_expected_filter_paths(PlannerInfo *root, RelOptInfo *rel) if (basepaths == NIL) return; + combinations = find_bloom_filter_combinations(root, rel); + if (combinations == NIL) + return; + /* - * Generate all combinations of the interesting filters. We do that by - * iterating 1 to (2^n-1), which generates all bitmask in between. Those - * are the subsets. - * - * XXX This is a good demonstration why we need to keep the number of - * filters low - * - * XXX Maybe we should also stop adding filters once the other filters - * already eliminate enought tuples. Say, we know F1 alone eliminates 99% - * tuples. Does it make sense to also consider [F1,F2]? Probably not. We - * could track "maximum" sets, and reject combinations containing one of - * those. We'd need to generate sets of increasing size, the iteration - * does not do that. But that's not hard. + * For each combination, build a fresh path from each eligible base path + * via that base path's real constructor (except IndexPath, see the + * function comment above). */ - for (combo = 1; combo < ((uint32) 1 << nfilters); combo++) + foreach_ptr(List, combo, combinations) { - List *subset = NIL; - Relids used = NULL; - bool overlap = false; - int i = 0; - ListCell *lcf; - - foreach(lcf, filters) + foreach_ptr(Path, base, basepaths) { - if (combo & ((uint32) 1 << i)) - subset = lappend(subset, lfirst(lcf)); - i++; - } + Path *newpath = NULL; - /* - * Skip combinations that mix filters with overlapping build sides. - * Such filters would be sourced from joins that share build - * relations, so their selectivities are not independent and applying - * them together at the same scan wrong - the result would be correct, - * but the estimates would get smaller. - */ - foreach(lcf, subset) - { - ExpectedFilter *f = (ExpectedFilter *) lfirst(lcf); - - if (bms_overlap(used, f->build_relids)) + switch (nodeTag(base)) { - overlap = true; - break; + case T_Path: + if (base->pathtype == T_SeqScan) + newpath = create_seqscan_path(root, rel, NULL, + base->parallel_workers, + combo); + else if (base->pathtype == T_SampleScan) + newpath = create_samplescan_path(root, rel, NULL, + combo); + break; + case T_IndexPath: + newpath = create_filtered_scan_path(root, base, combo); + break; + case T_BitmapHeapPath: + newpath = (Path *) + create_bitmap_heap_path(root, rel, + ((BitmapHeapPath *) base)->bitmapqual, + NULL, 1.0, + base->parallel_workers, + combo); + break; + case T_TidPath: + newpath = (Path *) + create_tidscan_path(root, rel, + ((TidPath *) base)->tidquals, + NULL, combo); + break; + case T_TidRangePath: + newpath = (Path *) + create_tidrangescan_path(root, rel, + ((TidRangePath *) base)->tidrangequals, + NULL, base->parallel_workers, + combo); + break; + default: + break; } - used = bms_add_members(used, f->build_relids); - } - bms_free(used); - - if (overlap) - { - list_free(subset); - continue; - } - /* - * All filtered paths for this combo share the same expected_filters - * list. That's safe: the list is never modified, and add_path() only - * ever frees the Path node itself, not its expected_filters. - */ - foreach(lc, basepaths) - { - Path *base = (Path *) lfirst(lc); - Path *newpath; - - newpath = create_filtered_scan_path(root, base, subset); if (newpath != NULL) add_path(rel, newpath); } } } +/* + * expected_filter_selectivity_cmp + * list_sort comparator ordering ExpectedFilters by ascending selectivity, + * i.e. the most selective filter (smallest surviving fraction) first. + */ +static int +expected_filter_selectivity_cmp(const ListCell *a, const ListCell *b) +{ + Selectivity sa = ((ExpectedFilter *) lfirst(a))->selectivity; + Selectivity sb = ((ExpectedFilter *) lfirst(b))->selectivity; + + if (sa < sb) + return -1; + if (sa > sb) + return 1; + return 0; +} + +/* + * find_bloom_filter_combinations + * Find the interesting Bloom filters rel could receive from a hash join + * above it (see find_interesting_bloom_filters), and return the combination + * of them worth building a path for, as a list of lists of ExpectedFilter. + * + * Following the "apply all candidates simultaneously" heuristic (Heuristic 4 + * of the bottom-up Bloom filter paper), we build a single combination that + * applies as many candidates as possible at once. We take the interesting + * filters most selective first and add each one whose build side does not + * overlap the build sides already chosen, stopping once + * bloom_filter_pushdown_max filters have been collected. + * + * Overlapping build sides are skipped because such filters would be sourced + * from joins sharing build relations, so their selectivities are not + * independent and applying them together would under-estimate the surviving + * rows. On such a conflict we keep the more selective filter, since it sorts + * first. bloom_filter_pushdown_max bounds how many filters are applied at + * once, which bounds the per-tuple probe cost. + * + * The result is a list of combinations (each a list of ExpectedFilter); this + * heuristic produces at most one combination. + * + * Note: A CustomScan provider can call it directly from its own + * set_rel_pathlist_hook (see generate_expected_filter_paths above). + */ +List * +find_bloom_filter_combinations(PlannerInfo *root, RelOptInfo *rel) +{ + List *filters = find_interesting_bloom_filters(root, rel); + List *sorted; + List *combo = NIL; + Relids used = NULL; + ListCell *lc; + + if (filters == NIL) + return NIL; + + /* + * Consider the most selective filters first, so that on a build-side + * conflict we keep the one that discards more rows, and so that the + * bloom_filter_pushdown_max cap retains the most useful filters. Sort a + * copy: "filters" is cached on the RelOptInfo and must not be reordered. + */ + sorted = list_copy(filters); + list_sort(sorted, expected_filter_selectivity_cmp); + + foreach(lc, sorted) + { + ExpectedFilter *f = (ExpectedFilter *) lfirst(lc); + + if (list_length(combo) >= bloom_filter_pushdown_max) + break; + + /* skip a filter whose build side overlaps one already chosen */ + if (bms_overlap(used, f->build_relids)) + continue; + + combo = lappend(combo, f); + used = bms_add_members(used, f->build_relids); + } + + list_free(sorted); + bms_free(used); + + if (combo == NIL) + return NIL; + + return list_make1(combo); +} + /* * set_tablesample_rel_size * Set size estimates for a sampled relation @@ -1674,7 +1707,7 @@ set_tablesample_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry * required_outer = rel->lateral_relids; /* Consider sampled scan */ - path = create_samplescan_path(root, rel, required_outer); + path = create_samplescan_path(root, rel, required_outer, NIL); /* * If the sampling method does not support repeatable scans, we must avoid @@ -5690,7 +5723,7 @@ create_partial_bitmap_paths(PlannerInfo *root, RelOptInfo *rel, return; add_partial_path(rel, (Path *) create_bitmap_heap_path(root, rel, - bitmapqual, rel->lateral_relids, 1.0, parallel_workers)); + bitmapqual, rel->lateral_relids, 1.0, parallel_workers, NIL)); } /* diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index 218a8634226..7569b880577 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -176,11 +176,11 @@ bool enable_async_append = true; double bloom_filter_pushdown_threshold = 0.3; /* - * Upper bound on the number of distinct interesting Bloom filters considered - * for a single scan relation. This bounds the number of additional paths - * generated per scan (the planner enumerates non-empty subsets of the - * interesting filters, i.e. up to 2^bloom_filter_pushdown_max - 1 extra - * paths per base scan path). + * Upper bound on the number of interesting Bloom filters that may be + * combined into a single filter-aware scan path variant for a base + * relation. This does not limit how many candidate filters are considered + * (see bloom_filter_pushdown_threshold) -- only how large a combination of + * them the planner is willing to build a path for. */ int bloom_filter_pushdown_max = 3; @@ -189,8 +189,8 @@ int bloom_filter_pushdown_max = 3; * side. Bloom filters over larger joins are unlikely to be worthwhile and * enumerating them would inflate planning time, so we keep this small. This * bounds the *size* of each candidate build side, which is distinct from - * bloom_filter_pushdown_max that bounds how many interesting filters are - * ultimately kept per probe relation. + * bloom_filter_pushdown_max, which bounds how many filters are applied + * together in a single scan. */ int bloom_filter_pushdown_max_build_relids = 3; diff --git a/src/backend/optimizer/path/indxpath.c b/src/backend/optimizer/path/indxpath.c index 3f5d4fa3182..a6c9a41ead1 100644 --- a/src/backend/optimizer/path/indxpath.c +++ b/src/backend/optimizer/path/indxpath.c @@ -344,7 +344,7 @@ create_index_paths(PlannerInfo *root, RelOptInfo *rel) bitmapqual = choose_bitmap_and(root, rel, bitindexpaths); bpath = create_bitmap_heap_path(root, rel, bitmapqual, - rel->lateral_relids, 1.0, 0); + rel->lateral_relids, 1.0, 0, NIL); add_path(rel, (Path *) bpath); /* create a partial bitmap heap path */ @@ -411,7 +411,7 @@ create_index_paths(PlannerInfo *root, RelOptInfo *rel) required_outer = PATH_REQ_OUTER(bitmapqual); loop_count = get_loop_count(root, rel->relid, required_outer); bpath = create_bitmap_heap_path(root, rel, bitmapqual, - required_outer, loop_count, 0); + required_outer, loop_count, 0, NIL); add_path(rel, (Path *) bpath); } } diff --git a/src/backend/optimizer/path/tidpath.c b/src/backend/optimizer/path/tidpath.c index 18a3654720b..83b261542ea 100644 --- a/src/backend/optimizer/path/tidpath.c +++ b/src/backend/optimizer/path/tidpath.c @@ -468,7 +468,7 @@ BuildParameterizedTidPaths(PlannerInfo *root, RelOptInfo *rel, List *clauses) required_outer = bms_del_member(required_outer, rel->relid); add_path(rel, (Path *) create_tidscan_path(root, rel, tidquals, - required_outer)); + required_outer, NIL)); } } @@ -520,7 +520,7 @@ create_tidscan_paths(PlannerInfo *root, RelOptInfo *rel) Relids required_outer = rel->lateral_relids; add_path(rel, (Path *) create_tidscan_path(root, rel, tidquals, - required_outer)); + required_outer, NIL)); /* * When the qual is CurrentOfExpr, the path that we just added is the @@ -554,7 +554,7 @@ create_tidscan_paths(PlannerInfo *root, RelOptInfo *rel) add_path(rel, (Path *) create_tidrangescan_path(root, rel, tidrangequals, required_outer, - 0)); + 0, NIL)); /* If appropriate, consider parallel tid range scan. */ if (rel->consider_parallel && required_outer == NULL) @@ -569,7 +569,7 @@ create_tidscan_paths(PlannerInfo *root, RelOptInfo *rel) rel, tidrangequals, required_outer, - parallel_workers)); + parallel_workers, NIL)); } } diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 6de8ffa8b03..eb853d267b6 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -7179,7 +7179,7 @@ plan_cluster_use_sort(Oid tableOid, Oid indexOid) comparisonCost = 2.0 * (indexExprCost.startup + indexExprCost.per_tuple); /* Estimate the cost of seq scan + sort */ - seqScanPath = create_seqscan_path(root, rel, NULL, 0); + seqScanPath = create_seqscan_path(root, rel, NULL, 0, NIL); cost_sort(&seqScanAndSortPath, root, NIL, seqScanPath->disabled_nodes, seqScanPath->total_cost, rel->tuples, rel->reltarget->width, diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c index 2e7c9c8bb18..ff4b2abd20c 100644 --- a/src/backend/optimizer/util/pathnode.c +++ b/src/backend/optimizer/util/pathnode.c @@ -459,72 +459,89 @@ expected_filters_selectivity(List *filters) return sel; } +/* + * Estimated cost of probing one Bloom filter for one tuple. A probe (a few + * hash evaluations and bit tests) is cheaper than a full hash-table lookup, + * so we charge less than cpu_operator_cost per probe; the bottom-up Bloom + * filter paper likewise sets its per-filter probe cost below a hash probe. + */ +#define BLOOM_FILTER_PROBE_COST (cpu_operator_cost * 0.5) + +/* + * apply_expected_filters + * Adjust an already-costed path to account for probing the given set of + * Bloom filters against the tuples it produces, and record them as the + * path's expected_filters. + * + * This is the generic, not-smart costing fallback. Filters are probed most + * selective first (see find_bloom_filter_combinations and, at execution, + * ExecInitBloomFilters), and a tuple rejected by one filter is not probed by + * the later ones. So the expected number of probes per input tuple is the + * sum, over the filters, of the fraction of tuples that survive all earlier + * filters -- far fewer than one probe per filter once the filters are + * selective. We charge BLOOM_FILTER_PROBE_COST per probe, on top of shrinking + * the row estimate by the filters' combined selectivity. + * + * Callers that can do something smarter (e.g. a CustomScan skipping whole row + * groups via a zone map) should cost themselves instead of using this. + */ +void +apply_expected_filters(Path *path, List *filters) +{ + double probes = 0.0; + double surviving = 1.0; + ListCell *lc; + + if (filters == NIL) + return; + + foreach(lc, filters) + { + ExpectedFilter *f = (ExpectedFilter *) lfirst(lc); + + probes += surviving; + surviving *= f->selectivity; + } + + path->total_cost += probes * BLOOM_FILTER_PROBE_COST * path->rows; + path->rows = clamp_row_est(path->rows * surviving); + path->expected_filters = filters; +} + /* * create_filtered_scan_path - * Build a copy of a base-relation scan path that additionally expects the - * given set of pushed-down Bloom filters. + * Build a copy of an IndexPath that additionally expects the given set of + * pushed-down Bloom filters. * * The clone shares all substructure with the original path (parent, - * pathtarget, clauses, etc.); only the rows estimate is reduced to reflect - * the filters' combined selectivity, and expected_filters is set. This is - * safe because create_plan() treats the clone identically to the original - * (it ignores expected_filters), and add_path() may freely pfree the clone. - * - * Only the plain scan path node types that can receive a pushed-down filter - * are supported (matching find_bloom_filter_recipient in createplan.c). - * Returns NULL for unsupported path types. + * pathtarget, indexclauses, pathkeys, etc.); we reuse the source path's + * already-derived indexclauses/pathkeys and its cost rather than rebuilding + * them, and then apply_expected_filters() makes the same cost/rows adjustment + * core makes for every other scan type -- charging the per-tuple probe cost + * and shrinking the row estimate. This is safe because create_plan() treats + * the clone identically to the original (it ignores expected_filters), and + * add_path() may freely pfree the clone. * - * XXX This should probably adjust the CPU cost in some way. It assumes the - * filter checks are free, which does not seem right. + * Other base-relation scan path types are built directly by their real + * constructors with the filters passed in (see create_seqscan_path, + * create_bitmap_heap_path, etc.); IndexPath is the exception because + * create_index_path() would re-derive its indexclauses/pathkeys from scratch + * -- see the comment on the T_IndexScan/T_IndexOnlyScan case in + * reparameterize_path() for the same tradeoff made there. */ Path * create_filtered_scan_path(PlannerInfo *root, Path *subpath, List *filters) { - Path *newpath; - size_t sz; + IndexPath *newpath; - switch (nodeTag(subpath)) - { - case T_Path: - /* plain seqscan/samplescan etc. */ - sz = sizeof(Path); - break; - case T_IndexPath: - sz = sizeof(IndexPath); - break; - case T_BitmapHeapPath: - sz = sizeof(BitmapHeapPath); - break; - case T_TidPath: - sz = sizeof(TidPath); - break; - case T_TidRangePath: - sz = sizeof(TidRangePath); - break; - case T_CustomPath: - - /* - * A base-relation CustomScan provider that advertised - * CUSTOMPATH_SUPPORT_BLOOM_FILTERS can receive a pushed-down - * filter and apply it in its own scan loop - * .generate_expected_filter_paths() only offers such paths here, - * so we need not re-check the flag. - */ - sz = sizeof(CustomPath); - break; - default: - /* unsupported scan path type */ - return NULL; - } + Assert(IsA(subpath, IndexPath)); - newpath = (Path *) palloc(sz); - memcpy(newpath, subpath, sz); + newpath = (IndexPath *) palloc(sizeof(IndexPath)); + memcpy(newpath, subpath, sizeof(IndexPath)); - newpath->expected_filters = filters; - newpath->rows = clamp_row_est(subpath->rows * - expected_filters_selectivity(filters)); + apply_expected_filters(&newpath->path, filters); - return newpath; + return (Path *) newpath; } /* @@ -1254,7 +1271,8 @@ add_partial_path_precheck(RelOptInfo *parent_rel, int disabled_nodes, */ Path * create_seqscan_path(PlannerInfo *root, RelOptInfo *rel, - Relids required_outer, int parallel_workers) + Relids required_outer, int parallel_workers, + List *filters) { Path *pathnode = makeNode(Path); @@ -1269,6 +1287,7 @@ create_seqscan_path(PlannerInfo *root, RelOptInfo *rel, pathnode->pathkeys = NIL; /* seqscan has unordered result */ cost_seqscan(pathnode, root, rel, pathnode->param_info); + apply_expected_filters(pathnode, filters); return pathnode; } @@ -1278,7 +1297,8 @@ create_seqscan_path(PlannerInfo *root, RelOptInfo *rel, * Creates a path node for a sampled table scan. */ Path * -create_samplescan_path(PlannerInfo *root, RelOptInfo *rel, Relids required_outer) +create_samplescan_path(PlannerInfo *root, RelOptInfo *rel, + Relids required_outer, List *filters) { Path *pathnode = makeNode(Path); @@ -1293,6 +1313,7 @@ create_samplescan_path(PlannerInfo *root, RelOptInfo *rel, Relids required_outer pathnode->pathkeys = NIL; /* samplescan has unordered result */ cost_samplescan(pathnode, root, rel, pathnode->param_info); + apply_expected_filters(pathnode, filters); return pathnode; } @@ -1381,7 +1402,8 @@ create_bitmap_heap_path(PlannerInfo *root, Path *bitmapqual, Relids required_outer, double loop_count, - int parallel_degree) + int parallel_degree, + List *filters) { BitmapHeapPath *pathnode = makeNode(BitmapHeapPath); @@ -1400,6 +1422,7 @@ create_bitmap_heap_path(PlannerInfo *root, cost_bitmap_heap_scan(&pathnode->path, root, rel, pathnode->path.param_info, bitmapqual, loop_count); + apply_expected_filters(&pathnode->path, filters); return pathnode; } @@ -1514,7 +1537,7 @@ create_bitmap_or_path(PlannerInfo *root, */ TidPath * create_tidscan_path(PlannerInfo *root, RelOptInfo *rel, List *tidquals, - Relids required_outer) + Relids required_outer, List *filters) { TidPath *pathnode = makeNode(TidPath); @@ -1532,6 +1555,7 @@ create_tidscan_path(PlannerInfo *root, RelOptInfo *rel, List *tidquals, cost_tidscan(&pathnode->path, root, rel, tidquals, pathnode->path.param_info); + apply_expected_filters(&pathnode->path, filters); return pathnode; } @@ -1544,7 +1568,7 @@ create_tidscan_path(PlannerInfo *root, RelOptInfo *rel, List *tidquals, TidRangePath * create_tidrangescan_path(PlannerInfo *root, RelOptInfo *rel, List *tidrangequals, Relids required_outer, - int parallel_workers) + int parallel_workers, List *filters) { TidRangePath *pathnode = makeNode(TidRangePath); @@ -1562,6 +1586,7 @@ create_tidrangescan_path(PlannerInfo *root, RelOptInfo *rel, cost_tidrangescan(&pathnode->path, root, rel, tidrangequals, pathnode->path.param_info); + apply_expected_filters(&pathnode->path, filters); return pathnode; } @@ -4176,9 +4201,9 @@ reparameterize_path(PlannerInfo *root, Path *path, switch (path->pathtype) { case T_SeqScan: - return create_seqscan_path(root, rel, required_outer, 0); + return create_seqscan_path(root, rel, required_outer, 0, NIL); case T_SampleScan: - return create_samplescan_path(root, rel, required_outer); + return create_samplescan_path(root, rel, required_outer, NIL); case T_IndexScan: case T_IndexOnlyScan: { @@ -4206,7 +4231,7 @@ reparameterize_path(PlannerInfo *root, Path *path, rel, bpath->bitmapqual, required_outer, - loop_count, 0); + loop_count, 0, NIL); } case T_SubqueryScan: { diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat index 01b9ee5b1cf..2fbe7f39b44 100644 --- a/src/backend/utils/misc/guc_parameters.dat +++ b/src/backend/utils/misc/guc_parameters.dat @@ -381,8 +381,8 @@ }, { name => 'bloom_filter_pushdown_max', type => 'int', context => 'PGC_USERSET', group => 'QUERY_TUNING_OTHER', - short_desc => 'Maximum number of pushed-down hash join bloom filters considered per scan.', - long_desc => 'Bounds how many interesting bloom filters the planner enumerates subsets of when building filter-aware scan paths.', + short_desc => 'Maximum number of hash join bloom filters combined into a single pushed-down filter set.', + long_desc => 'Bounds how many interesting bloom filters the planner will combine together when building filter-aware scan path variants for a base relation.', flags => 'GUC_EXPLAIN', variable => 'bloom_filter_pushdown_max', boot_val => '3', diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index 9080e9e0784..fe89e5a275c 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -486,8 +486,8 @@ # - Other Planner Options - -#bloom_filter_pushdown_max = 3 # range 0-10 -#bloom_filter_pushdown_threshold = 0.3 # range 0.0-1.0 +#bloom_filter_pushdown_max = 3 # range 0-10 +#bloom_filter_pushdown_threshold = 0.3 # range 0.0-1.0 #bloom_filter_pushdown_max_build_relids = 3 # range 1-100 #bloom_filter_pushdown_max_build_sets = 100 # range 1-INT_MAX #default_statistics_target = 100 # range 1-10000 diff --git a/src/include/lib/bloomfilter.h b/src/include/lib/bloomfilter.h index 9277087acb6..9a79e83e7d9 100644 --- a/src/include/lib/bloomfilter.h +++ b/src/include/lib/bloomfilter.h @@ -17,6 +17,10 @@ typedef struct bloom_filter bloom_filter; extern bloom_filter *bloom_create(int64 total_elems, int bloom_work_mem, uint64 seed); +extern bloom_filter *bloom_create_custom(int64 total_elems, int bloom_work_mem, + uint64 min_bitset_bytes, + int max_hash_funcs, + uint64 seed); extern void bloom_free(bloom_filter *filter); extern void bloom_add_element(bloom_filter *filter, unsigned char *elem, size_t len); diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 44c72e238cf..9fc7a11313c 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -1070,6 +1070,16 @@ typedef struct RelOptInfo struct Path *cheapest_total_path; List *cheapest_parameterized_paths; + /* + * Cache of the interesting Bloom filters this rel could receive from a + * hash join above it. Computed lazily and reused, since both core path + * generation and a CustomScan provider may ask for it, and recomputing it + * is not free. A separate "valid" flag distinguishes "not yet computed" + * from "computed, no filters". + */ + List *bloom_filters pg_node_attr(read_write_ignore); + bool bloom_filters_valid pg_node_attr(read_write_ignore); + /* * parameterization information needed for both base rels and join rels * (see also lateral_vars and lateral_referencers) diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h index f747f204184..97ff59cacda 100644 --- a/src/include/optimizer/pathnode.h +++ b/src/include/optimizer/pathnode.h @@ -66,13 +66,15 @@ extern bool add_partial_path_precheck(RelOptInfo *parent_rel, extern bool expected_filters_equal(List *a, List *b); extern double expected_filters_selectivity(List *filters); +extern void apply_expected_filters(Path *path, List *filters); extern Path *create_filtered_scan_path(PlannerInfo *root, Path *subpath, List *filters); extern Path *create_seqscan_path(PlannerInfo *root, RelOptInfo *rel, - Relids required_outer, int parallel_workers); + Relids required_outer, int parallel_workers, + List *filters); extern Path *create_samplescan_path(PlannerInfo *root, RelOptInfo *rel, - Relids required_outer); + Relids required_outer, List *filters); extern IndexPath *create_index_path(PlannerInfo *root, IndexOptInfo *index, List *indexclauses, @@ -89,7 +91,8 @@ extern BitmapHeapPath *create_bitmap_heap_path(PlannerInfo *root, Path *bitmapqual, Relids required_outer, double loop_count, - int parallel_degree); + int parallel_degree, + List *filters); extern BitmapAndPath *create_bitmap_and_path(PlannerInfo *root, RelOptInfo *rel, List *bitmapquals); @@ -97,12 +100,14 @@ extern BitmapOrPath *create_bitmap_or_path(PlannerInfo *root, RelOptInfo *rel, List *bitmapquals); extern TidPath *create_tidscan_path(PlannerInfo *root, RelOptInfo *rel, - List *tidquals, Relids required_outer); + List *tidquals, Relids required_outer, + List *filters); extern TidRangePath *create_tidrangescan_path(PlannerInfo *root, RelOptInfo *rel, List *tidrangequals, Relids required_outer, - int parallel_workers); + int parallel_workers, + List *filters); extern AppendPath *create_append_path(PlannerInfo *root, RelOptInfo *rel, AppendPathInput input, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index b153df758b3..2ec8a3bf928 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -61,6 +61,7 @@ extern PGDLLIMPORT join_search_hook_type join_search_hook; extern RelOptInfo *make_one_rel(PlannerInfo *root, List *joinlist); extern RelOptInfo *standard_join_search(PlannerInfo *root, int levels_needed, List *initial_rels); +extern List *find_bloom_filter_combinations(PlannerInfo *root, RelOptInfo *rel); extern void generate_gather_paths(PlannerInfo *root, RelOptInfo *rel, bool override_rows); diff --git a/src/test/modules/test_bloom_customscan/expected/test_bloom_customscan.out b/src/test/modules/test_bloom_customscan/expected/test_bloom_customscan.out index 8e784209901..11eea4b134e 100644 --- a/src/test/modules/test_bloom_customscan/expected/test_bloom_customscan.out +++ b/src/test/modules/test_bloom_customscan/expected/test_bloom_customscan.out @@ -107,8 +107,98 @@ SELECT test_bloom_cs_rejected_rows() > 0 AS filter_rejected_rows; t (1 row) +-- Two independent joins: the provider should offer, and the planner should +-- choose, a CustomPath expecting a combined filter from both joins. +-- cpu_operator_cost is lowered so combining is robustly cheaper rather than a +-- near-tie. +CREATE TABLE cs_wide_a (id int, label text); +CREATE TABLE cs_wide_b (id int, label text); +INSERT INTO cs_wide_a SELECT g, 'a' || g FROM generate_series(1, 400) g; +INSERT INTO cs_wide_b SELECT g, 'b' || g FROM generate_series(1, 400) g; +ANALYZE cs_wide_a; +ANALYZE cs_wide_b; +SET test_bloom_customscan.enable = on; +SET enable_seqscan = off; +SET cpu_operator_cost = 0.0001; +EXPLAIN (COSTS OFF, VERBOSE) +SELECT count(*) FROM cs_fact f + JOIN cs_wide_a wa ON f.a = wa.id + JOIN cs_wide_b wb ON f.b = wb.id; + QUERY PLAN +---------------------------------------------------------------------------------- + Aggregate + Output: count(*) + -> Hash Join + Hash Cond: (f.b = wb.id) + -> Hash Join + Output: f.b + Hash Cond: (f.a = wa.id) + -> Custom Scan (TestBloomCustomScan) on public.cs_fact f + Output: f.a, f.b + -> Hash + Output: wa.id + Bloom Filter 1 + -> Custom Scan (TestBloomCustomScan) on public.cs_wide_a wa + Output: wa.id + -> Hash + Output: wb.id + Bloom Filter 2 + -> Custom Scan (TestBloomCustomScan) on public.cs_wide_b wb + Output: wb.id +(19 rows) + +SELECT test_bloom_cs_reset(); + test_bloom_cs_reset +--------------------- + +(1 row) + +SELECT count(*) FROM cs_fact f + JOIN cs_wide_a wa ON f.a = wa.id + JOIN cs_wide_b wb ON f.b = wb.id; + count +------- + 8000 +(1 row) + +SELECT test_bloom_cs_rejected_rows() > 0 AS filter_rejected_rows; + filter_rejected_rows +---------------------- + t +(1 row) + +-- Correctness: the result must be identical with and without the filters. +SET test_bloom_customscan.enable = on; +SET enable_seqscan = off; +CREATE TEMP TABLE r_cs2 AS + SELECT f.a, count(*) AS n FROM cs_fact f + JOIN cs_wide_a wa ON f.a = wa.id + JOIN cs_wide_b wb ON f.b = wb.id + GROUP BY f.a; +SET test_bloom_customscan.enable = off; +SET enable_seqscan = on; +CREATE TEMP TABLE r_plain2 AS + SELECT f.a, count(*) AS n FROM cs_fact f + JOIN cs_wide_a wa ON f.a = wa.id + JOIN cs_wide_b wb ON f.b = wb.id + GROUP BY f.a; +SELECT count(*) AS cs_minus_plain + FROM (SELECT * FROM r_cs2 EXCEPT SELECT * FROM r_plain2) x; + cs_minus_plain +---------------- + 0 +(1 row) + +SELECT count(*) AS plain_minus_cs + FROM (SELECT * FROM r_plain2 EXCEPT SELECT * FROM r_cs2) x; + plain_minus_cs +---------------- + 0 +(1 row) + -- cleanup +RESET cpu_operator_cost; SET test_bloom_customscan.enable = off; SET enable_seqscan = on; -DROP TABLE cs_fact, cs_dim; +DROP TABLE cs_fact, cs_dim, cs_wide_a, cs_wide_b; DROP EXTENSION test_bloom_customscan; diff --git a/src/test/modules/test_bloom_customscan/sql/test_bloom_customscan.sql b/src/test/modules/test_bloom_customscan/sql/test_bloom_customscan.sql index daae84a531a..8da2e0789e6 100644 --- a/src/test/modules/test_bloom_customscan/sql/test_bloom_customscan.sql +++ b/src/test/modules/test_bloom_customscan/sql/test_bloom_customscan.sql @@ -59,8 +59,57 @@ SELECT count(*) FROM cs_fact f JOIN cs_dim d ON f.a = d.id AND f.b = d.id2; SELECT test_bloom_cs_perkey_built() AS perkey_filters_built; SELECT test_bloom_cs_rejected_rows() > 0 AS filter_rejected_rows; +-- Two independent joins: the provider should offer, and the planner should +-- choose, a CustomPath expecting a combined filter from both joins. +-- cpu_operator_cost is lowered so combining is robustly cheaper rather than a +-- near-tie. +CREATE TABLE cs_wide_a (id int, label text); +CREATE TABLE cs_wide_b (id int, label text); +INSERT INTO cs_wide_a SELECT g, 'a' || g FROM generate_series(1, 400) g; +INSERT INTO cs_wide_b SELECT g, 'b' || g FROM generate_series(1, 400) g; +ANALYZE cs_wide_a; +ANALYZE cs_wide_b; + +SET test_bloom_customscan.enable = on; +SET enable_seqscan = off; +SET cpu_operator_cost = 0.0001; + +EXPLAIN (COSTS OFF, VERBOSE) +SELECT count(*) FROM cs_fact f + JOIN cs_wide_a wa ON f.a = wa.id + JOIN cs_wide_b wb ON f.b = wb.id; + +SELECT test_bloom_cs_reset(); +SELECT count(*) FROM cs_fact f + JOIN cs_wide_a wa ON f.a = wa.id + JOIN cs_wide_b wb ON f.b = wb.id; +SELECT test_bloom_cs_rejected_rows() > 0 AS filter_rejected_rows; + +-- Correctness: the result must be identical with and without the filters. +SET test_bloom_customscan.enable = on; +SET enable_seqscan = off; +CREATE TEMP TABLE r_cs2 AS + SELECT f.a, count(*) AS n FROM cs_fact f + JOIN cs_wide_a wa ON f.a = wa.id + JOIN cs_wide_b wb ON f.b = wb.id + GROUP BY f.a; + +SET test_bloom_customscan.enable = off; +SET enable_seqscan = on; +CREATE TEMP TABLE r_plain2 AS + SELECT f.a, count(*) AS n FROM cs_fact f + JOIN cs_wide_a wa ON f.a = wa.id + JOIN cs_wide_b wb ON f.b = wb.id + GROUP BY f.a; + +SELECT count(*) AS cs_minus_plain + FROM (SELECT * FROM r_cs2 EXCEPT SELECT * FROM r_plain2) x; +SELECT count(*) AS plain_minus_cs + FROM (SELECT * FROM r_plain2 EXCEPT SELECT * FROM r_cs2) x; + -- cleanup +RESET cpu_operator_cost; SET test_bloom_customscan.enable = off; SET enable_seqscan = on; -DROP TABLE cs_fact, cs_dim; +DROP TABLE cs_fact, cs_dim, cs_wide_a, cs_wide_b; DROP EXTENSION test_bloom_customscan; diff --git a/src/test/modules/test_bloom_customscan/test_bloom_customscan.c b/src/test/modules/test_bloom_customscan/test_bloom_customscan.c index 99338c7d361..30f53ff45c7 100644 --- a/src/test/modules/test_bloom_customscan/test_bloom_customscan.c +++ b/src/test/modules/test_bloom_customscan/test_bloom_customscan.c @@ -64,6 +64,8 @@ typedef struct BloomCSScanState } BloomCSScanState; /* forward declarations */ +static void bloom_cs_add_path(RelOptInfo *rel, Cost startup_cost, + Cost total_cost, List *filters); static Plan *bloom_cs_plan_custom_path(PlannerInfo *root, RelOptInfo *rel, CustomPath *best_path, List *tlist, List *clauses, List *custom_plans); @@ -93,18 +95,57 @@ static const CustomExecMethods bloom_cs_exec_methods = { }; /* - * set_rel_pathlist_hook: offer a bloom-filter-capable CustomPath for plain + * bloom_cs_add_path + * Build and add one bloom-filter-capable CustomPath, optionally expecting + * the given set of pushed-down filters. We don't do anything smart with + * the filters ourselves, so apply_expected_filters() supplies the same + * generic per-tuple probe cost and row-count adjustment core uses for + * stock scan types. + */ +static void +bloom_cs_add_path(RelOptInfo *rel, Cost startup_cost, Cost total_cost, + List *filters) +{ + CustomPath *cpath = makeNode(CustomPath); + + cpath->path.pathtype = T_CustomScan; + cpath->path.parent = rel; + cpath->path.pathtarget = rel->reltarget; + cpath->path.param_info = NULL; + cpath->path.parallel_aware = false; + cpath->path.parallel_safe = false; + cpath->path.parallel_workers = 0; + cpath->path.rows = rel->rows; + cpath->path.startup_cost = startup_cost; + cpath->path.total_cost = total_cost; + cpath->path.pathkeys = NIL; + + cpath->flags = CUSTOMPATH_SUPPORT_BLOOM_FILTERS; + cpath->custom_paths = NIL; + cpath->custom_private = NIL; + cpath->methods = &bloom_cs_path_methods; + + apply_expected_filters(&cpath->path, filters); + + add_path(rel, (Path *) cpath); +} + +/* + * set_rel_pathlist_hook: offer bloom-filter-capable CustomPaths for plain * heap base relations. We copy the cost of the existing sequential scan path - * so join costing stays sane; the test forces the custom scan to be chosen - * with "SET enable_seqscan = off" (the CustomPath keeps disabled_nodes = 0). + * so join costing stays sane. + * + * Besides the plain, filter-oblivious path, we also offer one CustomPath per + * combination of interesting Bloom filters this rel could receive from a + * hash join above it. */ static void bloom_cs_set_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, Index rti, RangeTblEntry *rte) { - CustomPath *cpath; Cost startup_cost = 0; Cost total_cost = 0; + List *combinations; ListCell *lc; if (prev_set_rel_pathlist_hook) @@ -132,25 +173,16 @@ bloom_cs_set_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, Index rti, } } - cpath = makeNode(CustomPath); - cpath->path.pathtype = T_CustomScan; - cpath->path.parent = rel; - cpath->path.pathtarget = rel->reltarget; - cpath->path.param_info = NULL; - cpath->path.parallel_aware = false; - cpath->path.parallel_safe = false; - cpath->path.parallel_workers = 0; - cpath->path.rows = rel->rows; - cpath->path.startup_cost = startup_cost; - cpath->path.total_cost = total_cost; - cpath->path.pathkeys = NIL; + bloom_cs_add_path(rel, startup_cost, total_cost, NIL); - cpath->flags = CUSTOMPATH_SUPPORT_BLOOM_FILTERS; - cpath->custom_paths = NIL; - cpath->custom_private = NIL; - cpath->methods = &bloom_cs_path_methods; + combinations = find_bloom_filter_combinations(root, rel); - add_path(rel, (Path *) cpath); + foreach(lc, combinations) + { + List *combo = (List *) lfirst(lc); + + bloom_cs_add_path(rel, startup_cost, total_cost, combo); + } } static Plan * diff --git a/src/test/regress/expected/hashjoin_bloom.out b/src/test/regress/expected/hashjoin_bloom.out index 793aff233a3..3ca58f850a1 100644 --- a/src/test/regress/expected/hashjoin_bloom.out +++ b/src/test/regress/expected/hashjoin_bloom.out @@ -113,13 +113,13 @@ JOIN bloom_multi_dim d ON (f.id1 = d.id1 AND f.id2 = d.id2) WHERE d.r < 0.5; QUERY PLAN -------------------------------------------------------------------------------------------------------------- - Hash Join (cost=25.98..685.58 rows=14970 width=53) (actual rows=14970.00 loops=1) + Hash Join (cost=25.98..723.08 rows=14970 width=53) (actual rows=14970.00 loops=1) Hash Cond: ((f.id1 = d.id1) AND (f.id2 = d.id2)) - -> Seq Scan on bloom_multi_fact f (cost=0.00..581.00 rows=14970 width=41) (actual rows=14970.00 loops=1) - Bloom Filter 1: keys=(id1, id2) expected=49.9% checked=29999 rejected=15030 (50.1%) + -> Seq Scan on bloom_multi_fact f (cost=0.00..618.50 rows=14970 width=41) (actual rows=15000.00 loops=1) + Bloom Filter 1: keys=(id1, id2) expected=49.9% checked=29999 rejected=15000 (50.0%) -> Hash (cost=18.50..18.50 rows=499 width=12) (actual rows=499.00 loops=1) Buckets: 1024 Batches: 1 Memory Usage: 30kB - Bloom Filter 1: bits=8388608 hashes=10 memory=1024kB checked=29999 rejected=15030 + Bloom Filter 1: bits=8192 hashes=4 memory=1kB checked=29999 rejected=15000 -> Seq Scan on bloom_multi_dim d (cost=0.00..18.50 rows=499 width=12) (actual rows=499.00 loops=1) Filter: (r < '0.5'::double precision) Rows Removed by Filter: 501 diff --git a/src/test/regress/expected/hashjoin_bloom_snowflake.out b/src/test/regress/expected/hashjoin_bloom_snowflake.out index b06e42696d6..03aa81ca1a7 100644 --- a/src/test/regress/expected/hashjoin_bloom_snowflake.out +++ b/src/test/regress/expected/hashjoin_bloom_snowflake.out @@ -443,29 +443,29 @@ JOIN bloom_snowflake_multi_dim_1_2 d12 ON (d1.id12a = d12.a AND d1.id12b = d12.b WHERE d11.r < 0.45 AND d12.r < 0.55; QUERY PLAN ------------------------------------------------------------------------------------------------------------------------------------------------ - Hash Join (cost=76.27..2292.04 rows=24628 width=101) (actual rows=23530.00 loops=1) + Hash Join (cost=78.07..2418.84 rows=24628 width=101) (actual rows=23530.00 loops=1) Hash Cond: ((f.id1a = d1.a) AND (f.id1b = d1.b)) - -> Seq Scan on bloom_snowflake_multi_fact f (cost=0.00..2031.00 rows=24628 width=49) (actual rows=23531.00 loops=1) + -> Seq Scan on bloom_snowflake_multi_fact f (cost=0.00..2156.00 rows=24628 width=49) (actual rows=23531.00 loops=1) Bloom Filter 3: keys=(id1a, id1b) expected=24.6% checked=99999 rejected=76469 (76.5%) - -> Hash (cost=72.58..72.58 rows=246 width=52) (actual rows=234.00 loops=1) + -> Hash (cost=74.38..74.38 rows=246 width=52) (actual rows=234.00 loops=1) Buckets: 1024 Batches: 1 Memory Usage: 28kB - Bloom Filter 3: bits=8388608 hashes=10 memory=1024kB checked=99999 rejected=76469 - -> Hash Join (cost=52.00..72.58 rows=246 width=52) (actual rows=234.00 loops=1) + Bloom Filter 3: bits=8192 hashes=4 memory=1kB checked=99999 rejected=76469 + -> Hash Join (cost=52.00..74.38 rows=246 width=52) (actual rows=234.00 loops=1) Hash Cond: ((d1.id12a = d12.a) AND (d1.id12b = d12.b)) - -> Hash Join (cost=25.09..44.38 rows=246 width=40) (actual rows=234.00 loops=1) + -> Hash Join (cost=25.09..46.18 rows=246 width=40) (actual rows=235.00 loops=1) Hash Cond: ((d1.id11a = d11.a) AND (d1.id11b = d11.b)) - -> Seq Scan on bloom_snowflake_multi_dim_1 d1 (cost=0.00..18.00 rows=246 width=28) (actual rows=235.00 loops=1) + -> Seq Scan on bloom_snowflake_multi_dim_1 d1 (cost=0.00..19.80 rows=246 width=28) (actual rows=236.00 loops=1) Bloom Filter 1: keys=(id11a, id11b) expected=43.9% checked=996 rejected=569 (57.1%) - Bloom Filter 2: keys=(id12a, id12b) expected=56.1% checked=431 rejected=196 (45.5%) + Bloom Filter 2: keys=(id12a, id12b) expected=56.1% checked=431 rejected=195 (45.2%) -> Hash (cost=18.50..18.50 rows=439 width=12) (actual rows=440.00 loops=1) Buckets: 1024 Batches: 1 Memory Usage: 27kB - Bloom Filter 1: bits=8388608 hashes=10 memory=1024kB checked=996 rejected=569 + Bloom Filter 1: bits=8192 hashes=4 memory=1kB checked=996 rejected=569 -> Seq Scan on bloom_snowflake_multi_dim_1_1 d11 (cost=0.00..18.50 rows=439 width=12) (actual rows=440.00 loops=1) Filter: (r < '0.45'::double precision) Rows Removed by Filter: 560 -> Hash (cost=18.50..18.50 rows=561 width=12) (actual rows=562.00 loops=1) Buckets: 1024 Batches: 1 Memory Usage: 33kB - Bloom Filter 2: bits=8388608 hashes=10 memory=1024kB checked=431 rejected=196 + Bloom Filter 2: bits=8192 hashes=4 memory=1kB checked=431 rejected=195 -> Seq Scan on bloom_snowflake_multi_dim_1_2 d12 (cost=0.00..18.50 rows=561 width=12) (actual rows=562.00 loops=1) Filter: (r < '0.55'::double precision) Rows Removed by Filter: 438 @@ -480,13 +480,13 @@ JOIN bloom_snowflake_multi_dim_1_2 d12 ON (d1.id12a = d12.a AND d1.id12b = d12.b WHERE d11.r < 0.75 AND d12.r < 0.75; QUERY PLAN ------------------------------------------------------------------------------------------------------------------------------------------------ - Hash Join (cost=94.81..2542.85 rows=55564 width=101) (actual rows=55598.00 loops=1) + Hash Join (cost=94.81..2667.85 rows=55564 width=101) (actual rows=55598.00 loops=1) Hash Cond: ((f.id1a = d1.a) AND (f.id1b = d1.b)) - -> Seq Scan on bloom_snowflake_multi_fact f (cost=0.00..2031.00 rows=55564 width=49) (actual rows=55598.00 loops=1) - Bloom Filter 1: keys=(id1a, id1b) expected=55.6% checked=99999 rejected=44402 (44.4%) + -> Seq Scan on bloom_snowflake_multi_fact f (cost=0.00..2156.00 rows=55564 width=49) (actual rows=55694.00 loops=1) + Bloom Filter 1: keys=(id1a, id1b) expected=55.6% checked=99999 rejected=44306 (44.3%) -> Hash (cost=86.47..86.47 rows=556 width=52) (actual rows=555.00 loops=1) Buckets: 1024 Batches: 1 Memory Usage: 54kB - Bloom Filter 1: bits=8388608 hashes=10 memory=1024kB checked=99999 rejected=44402 + Bloom Filter 1: bits=8192 hashes=4 memory=1kB checked=99999 rejected=44306 -> Hash Join (cost=59.36..86.47 rows=556 width=52) (actual rows=555.00 loops=1) Hash Cond: ((d1.id12a = d12.a) AND (d1.id12b = d12.b)) -> Hash Join (cost=29.51..52.76 rows=734 width=40) (actual rows=737.00 loops=1) @@ -516,12 +516,12 @@ JOIN bloom_snowflake_multi_dim_2_2 d22 ON (d2.id22a = d22.a AND d2.id22b = d22.b WHERE d11.r < 0.75 AND d12.r < 0.75; QUERY PLAN ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- - Hash Join (cost=331.99..3197.32 rows=55564 width=153) (actual rows=55598.00 loops=1) + Hash Join (cost=331.99..3322.32 rows=55564 width=153) (actual rows=55598.00 loops=1) Hash Cond: ((f.id1a = d1.a) AND (f.id1b = d1.b)) - -> Hash Join (cost=237.18..2685.47 rows=55564 width=101) (actual rows=55598.00 loops=1) + -> Hash Join (cost=237.18..2810.47 rows=55564 width=101) (actual rows=55694.00 loops=1) Hash Cond: ((f.id2a = d2.a) AND (f.id2b = d2.b)) - -> Seq Scan on bloom_snowflake_multi_fact f (cost=0.00..2031.00 rows=55564 width=49) (actual rows=55598.00 loops=1) - Bloom Filter 1: keys=(id1a, id1b) expected=55.6% checked=100000 rejected=44402 (44.4%) + -> Seq Scan on bloom_snowflake_multi_fact f (cost=0.00..2156.00 rows=55564 width=49) (actual rows=55694.00 loops=1) + Bloom Filter 1: keys=(id1a, id1b) expected=55.6% checked=100000 rejected=44306 (44.3%) -> Hash (cost=222.18..222.18 rows=1000 width=52) (actual rows=1000.00 loops=1) Buckets: 1024 Batches: 1 Memory Usage: 91kB -> Merge Join (cost=170.18..222.18 rows=1000 width=52) (actual rows=1000.00 loops=1) @@ -541,7 +541,7 @@ WHERE d11.r < 0.75 AND d12.r < 0.75; -> Seq Scan on bloom_snowflake_multi_dim_2 d2 (cost=0.00..18.00 rows=1000 width=28) (actual rows=1000.00 loops=1) -> Hash (cost=86.47..86.47 rows=556 width=52) (actual rows=555.00 loops=1) Buckets: 1024 Batches: 1 Memory Usage: 54kB - Bloom Filter 1: bits=8388608 hashes=10 memory=1024kB checked=100000 rejected=44402 + Bloom Filter 1: bits=8192 hashes=4 memory=1kB checked=100000 rejected=44306 -> Hash Join (cost=59.36..86.47 rows=556 width=52) (actual rows=555.00 loops=1) Hash Cond: ((d1.id12a = d12.a) AND (d1.id12b = d12.b)) -> Hash Join (cost=29.51..52.76 rows=734 width=40) (actual rows=737.00 loops=1) diff --git a/src/test/regress/expected/hashjoin_bloom_star.out b/src/test/regress/expected/hashjoin_bloom_star.out index 94635408206..99503f7a8f1 100644 --- a/src/test/regress/expected/hashjoin_bloom_star.out +++ b/src/test/regress/expected/hashjoin_bloom_star.out @@ -482,25 +482,25 @@ JOIN bloom_star_multi_dim_7 d7 ON (f.id7a = d7.a AND f.id7b = d7.b) WHERE d1.r < 0.5; QUERY PLAN ---------------------------------------------------------------------------------------------------------------------------------------------------------- - Hash Join (cost=211.53..4474.11 rows=46900 width=173) (actual rows=46848.00 loops=1) + Hash Join (cost=211.53..4599.11 rows=46900 width=173) (actual rows=46848.00 loops=1) Hash Cond: ((f.id7a = d7.a) AND (f.id7b = d7.b)) - -> Hash Join (cost=180.53..4196.88 rows=46900 width=161) (actual rows=46848.00 loops=1) + -> Hash Join (cost=180.53..4321.89 rows=46900 width=161) (actual rows=46848.00 loops=1) Hash Cond: ((f.id6a = d6.a) AND (f.id6b = d6.b)) - -> Hash Join (cost=149.53..3919.66 rows=46900 width=149) (actual rows=46848.00 loops=1) + -> Hash Join (cost=149.53..4044.66 rows=46900 width=149) (actual rows=46848.00 loops=1) Hash Cond: ((f.id5a = d5.a) AND (f.id5b = d5.b)) - -> Hash Join (cost=118.53..3642.43 rows=46900 width=137) (actual rows=46848.00 loops=1) + -> Hash Join (cost=118.53..3767.43 rows=46900 width=137) (actual rows=46848.00 loops=1) Hash Cond: ((f.id4a = d4.a) AND (f.id4b = d4.b)) - -> Hash Join (cost=87.53..3365.21 rows=46900 width=125) (actual rows=46848.00 loops=1) + -> Hash Join (cost=87.53..3490.21 rows=46900 width=125) (actual rows=46848.00 loops=1) Hash Cond: ((f.id3a = d3.a) AND (f.id3b = d3.b)) - -> Hash Join (cost=56.53..3087.98 rows=46900 width=113) (actual rows=46848.00 loops=1) + -> Hash Join (cost=56.53..3212.98 rows=46900 width=113) (actual rows=46848.00 loops=1) Hash Cond: ((f.id2a = d2.a) AND (f.id2b = d2.b)) - -> Hash Join (cost=25.54..2810.76 rows=46900 width=101) (actual rows=46848.00 loops=1) + -> Hash Join (cost=25.54..2935.76 rows=46900 width=101) (actual rows=46848.00 loops=1) Hash Cond: ((f.id1a = d1.a) AND (f.id1b = d1.b)) - -> Seq Scan on bloom_star_multi_fact f (cost=0.00..2539.00 rows=46900 width=89) (actual rows=46848.00 loops=1) + -> Seq Scan on bloom_star_multi_fact f (cost=0.00..2664.00 rows=46900 width=89) (actual rows=46848.00 loops=1) Bloom Filter 1: keys=(id1a, id1b) expected=46.9% checked=99999 rejected=53152 (53.2%) -> Hash (cost=18.50..18.50 rows=469 width=12) (actual rows=470.00 loops=1) Buckets: 1024 Batches: 1 Memory Usage: 29kB - Bloom Filter 1: bits=8388608 hashes=10 memory=1024kB checked=99999 rejected=53152 + Bloom Filter 1: bits=8192 hashes=4 memory=1kB checked=99999 rejected=53152 -> Seq Scan on bloom_star_multi_dim_1 d1 (cost=0.00..18.50 rows=469 width=12) (actual rows=470.00 loops=1) Filter: (r < '0.5'::double precision) Rows Removed by Filter: 530 @@ -538,26 +538,26 @@ JOIN bloom_star_multi_dim_7 d7 ON (f.id7a = d7.a AND f.id7b = d7.b) WHERE d1.r < 0.4 AND d7.r < 0.5; QUERY PLAN ---------------------------------------------------------------------------------------------------------------------------------------------------------- - Hash Join (cost=204.78..3401.16 rows=17888 width=173) (actual rows=17864.00 loops=1) + Hash Join (cost=204.78..3573.04 rows=17888 width=173) (actual rows=17864.00 loops=1) Hash Cond: ((f.id7a = d7.a) AND (f.id7b = d7.b)) - -> Hash Join (cost=179.12..3281.60 rows=17888 width=161) (actual rows=17864.00 loops=1) + -> Hash Join (cost=179.12..3453.47 rows=17888 width=161) (actual rows=17899.00 loops=1) Hash Cond: ((f.id6a = d6.a) AND (f.id6b = d6.b)) - -> Hash Join (cost=148.12..3156.69 rows=17888 width=149) (actual rows=17864.00 loops=1) + -> Hash Join (cost=148.12..3328.56 rows=17888 width=149) (actual rows=17899.00 loops=1) Hash Cond: ((f.id5a = d5.a) AND (f.id5b = d5.b)) - -> Hash Join (cost=117.12..3031.77 rows=17888 width=137) (actual rows=17864.00 loops=1) + -> Hash Join (cost=117.12..3203.65 rows=17888 width=137) (actual rows=17899.00 loops=1) Hash Cond: ((f.id4a = d4.a) AND (f.id4b = d4.b)) - -> Hash Join (cost=86.12..2906.86 rows=17888 width=125) (actual rows=17864.00 loops=1) + -> Hash Join (cost=86.12..3078.74 rows=17888 width=125) (actual rows=17899.00 loops=1) Hash Cond: ((f.id3a = d3.a) AND (f.id3b = d3.b)) - -> Hash Join (cost=55.12..2781.95 rows=17888 width=113) (actual rows=17864.00 loops=1) + -> Hash Join (cost=55.12..2953.82 rows=17888 width=113) (actual rows=17899.00 loops=1) Hash Cond: ((f.id2a = d2.a) AND (f.id2b = d2.b)) - -> Hash Join (cost=24.12..2657.04 rows=17888 width=101) (actual rows=17864.00 loops=1) + -> Hash Join (cost=24.12..2828.91 rows=17888 width=101) (actual rows=17899.00 loops=1) Hash Cond: ((f.id1a = d1.a) AND (f.id1b = d1.b)) - -> Seq Scan on bloom_star_multi_fact f (cost=0.00..2539.00 rows=17888 width=89) (actual rows=17864.00 loops=1) + -> Seq Scan on bloom_star_multi_fact f (cost=0.00..2710.88 rows=17888 width=89) (actual rows=17899.00 loops=1) Bloom Filter 1: keys=(id1a, id1b) expected=37.5% checked=99999 rejected=62610 (62.6%) - Bloom Filter 2: keys=(id7a, id7b) expected=47.7% checked=37390 rejected=19526 (52.2%) + Bloom Filter 2: keys=(id7a, id7b) expected=47.7% checked=37390 rejected=19491 (52.1%) -> Hash (cost=18.50..18.50 rows=375 width=12) (actual rows=376.00 loops=1) Buckets: 1024 Batches: 1 Memory Usage: 25kB - Bloom Filter 1: bits=8388608 hashes=10 memory=1024kB checked=99999 rejected=62610 + Bloom Filter 1: bits=8192 hashes=4 memory=1kB checked=99999 rejected=62610 -> Seq Scan on bloom_star_multi_dim_1 d1 (cost=0.00..18.50 rows=375 width=12) (actual rows=376.00 loops=1) Filter: (r < '0.4'::double precision) Rows Removed by Filter: 624 @@ -578,7 +578,7 @@ WHERE d1.r < 0.4 AND d7.r < 0.5; -> Seq Scan on bloom_star_multi_dim_6 d6 (cost=0.00..16.00 rows=1000 width=12) (actual rows=1000.00 loops=1) -> Hash (cost=18.50..18.50 rows=477 width=12) (actual rows=478.00 loops=1) Buckets: 1024 Batches: 1 Memory Usage: 29kB - Bloom Filter 2: bits=8388608 hashes=10 memory=1024kB checked=37390 rejected=19526 + Bloom Filter 2: bits=8192 hashes=4 memory=1kB checked=37390 rejected=19491 -> Seq Scan on bloom_star_multi_dim_7 d7 (cost=0.00..18.50 rows=477 width=12) (actual rows=478.00 loops=1) Filter: (r < '0.5'::double precision) Rows Removed by Filter: 522 @@ -598,39 +598,39 @@ JOIN bloom_star_multi_dim_7 d7 ON (f.id7a = d7.a AND f.id7b = d7.b) WHERE d1.r < 0.3 AND d2.r < 0.35 AND d3.r < 0.4 AND d4.r < 0.45 AND d5.r < 0.5 AND d6.r < 0.55 AND d7.r < 0.6; QUERY PLAN ---------------------------------------------------------------------------------------------------------------------------------------------------------- - Hash Join (cost=176.01..2816.72 rows=283 width=173) (actual rows=292.00 loops=1) + Hash Join (cost=176.01..2991.80 rows=283 width=173) (actual rows=292.00 loops=1) Hash Cond: ((f.id7a = d7.a) AND (f.id7b = d7.b)) - -> Hash Join (cost=149.07..2787.14 rows=503 width=161) (actual rows=526.00 loops=1) + -> Hash Join (cost=149.07..2962.22 rows=503 width=161) (actual rows=526.00 loops=1) Hash Cond: ((f.id6a = d6.a) AND (f.id6b = d6.b)) - -> Hash Join (cost=122.51..2755.67 rows=936 width=149) (actual rows=923.00 loops=1) + -> Hash Join (cost=122.51..2930.75 rows=936 width=149) (actual rows=923.00 loops=1) Hash Cond: ((f.id5a = d5.a) AND (f.id5b = d5.b)) - -> Hash Join (cost=96.24..2719.92 rows=1806 width=137) (actual rows=1774.00 loops=1) + -> Hash Join (cost=96.24..2895.00 rows=1806 width=137) (actual rows=1774.00 loops=1) Hash Cond: ((f.id4a = d4.a) AND (f.id4b = d4.b)) - -> Hash Join (cost=71.02..2673.53 rows=4032 width=125) (actual rows=3997.00 loops=1) + -> Hash Join (cost=71.02..2848.61 rows=4032 width=125) (actual rows=3997.00 loops=1) Hash Cond: ((f.id3a = d3.a) AND (f.id3b = d3.b)) - -> Hash Join (cost=46.69..2628.03 rows=4032 width=113) (actual rows=3997.00 loops=1) + -> Hash Join (cost=46.69..2803.11 rows=4032 width=113) (actual rows=4005.00 loops=1) Hash Cond: ((f.id2a = d2.a) AND (f.id2b = d2.b)) - -> Hash Join (cost=22.95..2583.12 rows=4032 width=101) (actual rows=3997.00 loops=1) + -> Hash Join (cost=22.95..2758.20 rows=4032 width=101) (actual rows=4016.00 loops=1) Hash Cond: ((f.id1a = d1.a) AND (f.id1b = d1.b)) - -> Seq Scan on bloom_star_multi_fact f (cost=0.00..2539.00 rows=4032 width=89) (actual rows=3997.00 loops=1) + -> Seq Scan on bloom_star_multi_fact f (cost=0.00..2714.08 rows=4032 width=89) (actual rows=4016.00 loops=1) Bloom Filter 1: keys=(id1a, id1b) expected=29.7% checked=99991 rejected=70413 (70.4%) - Bloom Filter 2: keys=(id2a, id2b) expected=34.9% checked=29587 rejected=19399 (65.6%) - Bloom Filter 3: keys=(id3a, id3b) expected=38.9% checked=10188 rejected=6191 (60.8%) + Bloom Filter 2: keys=(id2a, id2b) expected=34.9% checked=29587 rejected=19372 (65.5%) + Bloom Filter 3: keys=(id3a, id3b) expected=38.9% checked=10215 rejected=6199 (60.7%) -> Hash (cost=18.50..18.50 rows=297 width=12) (actual rows=298.00 loops=1) Buckets: 1024 Batches: 1 Memory Usage: 21kB - Bloom Filter 1: bits=8388608 hashes=10 memory=1024kB checked=99991 rejected=70413 + Bloom Filter 1: bits=8192 hashes=4 memory=1kB checked=99991 rejected=70413 -> Seq Scan on bloom_star_multi_dim_1 d1 (cost=0.00..18.50 rows=297 width=12) (actual rows=298.00 loops=1) Filter: (r < '0.3'::double precision) Rows Removed by Filter: 702 -> Hash (cost=18.50..18.50 rows=349 width=12) (actual rows=350.00 loops=1) Buckets: 1024 Batches: 1 Memory Usage: 24kB - Bloom Filter 2: bits=8388608 hashes=10 memory=1024kB checked=29587 rejected=19399 + Bloom Filter 2: bits=8192 hashes=4 memory=1kB checked=29587 rejected=19372 -> Seq Scan on bloom_star_multi_dim_2 d2 (cost=0.00..18.50 rows=349 width=12) (actual rows=350.00 loops=1) Filter: (r < '0.35'::double precision) Rows Removed by Filter: 650 -> Hash (cost=18.50..18.50 rows=389 width=12) (actual rows=391.00 loops=1) Buckets: 1024 Batches: 1 Memory Usage: 25kB - Bloom Filter 3: bits=8388608 hashes=10 memory=1024kB checked=10188 rejected=6191 + Bloom Filter 3: bits=8192 hashes=4 memory=1kB checked=10215 rejected=6199 -> Seq Scan on bloom_star_multi_dim_3 d3 (cost=0.00..18.50 rows=389 width=12) (actual rows=391.00 loops=1) Filter: (r < '0.4'::double precision) Rows Removed by Filter: 609 @@ -671,67 +671,67 @@ JOIN bloom_star_multi_dim_7 d7 ON (f.id7a = d7.a AND f.id7b = d7.b) WHERE d1.r < 0.3 AND d2.r < 0.35 AND d3.r < 0.4 AND d4.r < 0.45 AND d5.r < 0.5 AND d6.r < 0.55 AND d7.r < 0.6; QUERY PLAN ---------------------------------------------------------------------------------------------------------------------------------------------------------- - Hash Join (cost=176.01..2725.42 rows=283 width=173) (actual rows=292.00 loops=1) + Hash Join (cost=176.01..2909.59 rows=283 width=173) (actual rows=292.00 loops=1) Hash Cond: ((f.id7a = d7.a) AND (f.id7b = d7.b)) - -> Hash Join (cost=149.07..2696.98 rows=283 width=161) (actual rows=292.00 loops=1) + -> Hash Join (cost=149.07..2881.16 rows=283 width=161) (actual rows=292.00 loops=1) Hash Cond: ((f.id6a = d6.a) AND (f.id6b = d6.b)) - -> Hash Join (cost=122.51..2668.94 rows=283 width=149) (actual rows=292.00 loops=1) + -> Hash Join (cost=122.51..2853.12 rows=283 width=149) (actual rows=292.00 loops=1) Hash Cond: ((f.id5a = d5.a) AND (f.id5b = d5.b)) - -> Hash Join (cost=96.24..2641.19 rows=283 width=137) (actual rows=292.00 loops=1) + -> Hash Join (cost=96.24..2825.37 rows=283 width=137) (actual rows=296.00 loops=1) Hash Cond: ((f.id4a = d4.a) AND (f.id4b = d4.b)) - -> Hash Join (cost=71.02..2614.48 rows=283 width=125) (actual rows=292.00 loops=1) + -> Hash Join (cost=71.02..2798.66 rows=283 width=125) (actual rows=296.00 loops=1) Hash Cond: ((f.id3a = d3.a) AND (f.id3b = d3.b)) - -> Hash Join (cost=46.69..2588.66 rows=283 width=113) (actual rows=292.00 loops=1) + -> Hash Join (cost=46.69..2772.84 rows=283 width=113) (actual rows=298.00 loops=1) Hash Cond: ((f.id2a = d2.a) AND (f.id2b = d2.b)) - -> Hash Join (cost=22.95..2563.44 rows=283 width=101) (actual rows=292.00 loops=1) + -> Hash Join (cost=22.95..2747.62 rows=283 width=101) (actual rows=299.00 loops=1) Hash Cond: ((f.id1a = d1.a) AND (f.id1b = d1.b)) - -> Seq Scan on bloom_star_multi_fact f (cost=0.00..2539.00 rows=283 width=89) (actual rows=292.00 loops=1) + -> Seq Scan on bloom_star_multi_fact f (cost=0.00..2723.18 rows=283 width=89) (actual rows=299.00 loops=1) Bloom Filter 1: keys=(id1a, id1b) expected=29.7% checked=99991 rejected=70413 (70.4%) - Bloom Filter 2: keys=(id2a, id2b) expected=34.9% checked=29587 rejected=19399 (65.6%) - Bloom Filter 3: keys=(id3a, id3b) expected=38.9% checked=10188 rejected=6191 (60.8%) - Bloom Filter 4: keys=(id4a, id4b) expected=44.8% checked=3997 rejected=2223 (55.6%) - Bloom Filter 5: keys=(id5a, id5b) expected=51.8% checked=1774 rejected=851 (48.0%) - Bloom Filter 6: keys=(id6a, id6b) expected=53.7% checked=923 rejected=397 (43.0%) - Bloom Filter 7: keys=(id7a, id7b) expected=56.3% checked=526 rejected=234 (44.5%) + Bloom Filter 2: keys=(id2a, id2b) expected=34.9% checked=29587 rejected=19372 (65.5%) + Bloom Filter 3: keys=(id3a, id3b) expected=38.9% checked=10215 rejected=6199 (60.7%) + Bloom Filter 4: keys=(id4a, id4b) expected=44.8% checked=4016 rejected=2234 (55.6%) + Bloom Filter 5: keys=(id5a, id5b) expected=51.8% checked=1782 rejected=848 (47.6%) + Bloom Filter 6: keys=(id6a, id6b) expected=53.7% checked=934 rejected=399 (42.7%) + Bloom Filter 7: keys=(id7a, id7b) expected=56.3% checked=535 rejected=236 (44.1%) -> Hash (cost=18.50..18.50 rows=297 width=12) (actual rows=298.00 loops=1) Buckets: 1024 Batches: 1 Memory Usage: 21kB - Bloom Filter 1: bits=8388608 hashes=10 memory=1024kB checked=99991 rejected=70413 + Bloom Filter 1: bits=8192 hashes=4 memory=1kB checked=99991 rejected=70413 -> Seq Scan on bloom_star_multi_dim_1 d1 (cost=0.00..18.50 rows=297 width=12) (actual rows=298.00 loops=1) Filter: (r < '0.3'::double precision) Rows Removed by Filter: 702 -> Hash (cost=18.50..18.50 rows=349 width=12) (actual rows=350.00 loops=1) Buckets: 1024 Batches: 1 Memory Usage: 24kB - Bloom Filter 2: bits=8388608 hashes=10 memory=1024kB checked=29587 rejected=19399 + Bloom Filter 2: bits=8192 hashes=4 memory=1kB checked=29587 rejected=19372 -> Seq Scan on bloom_star_multi_dim_2 d2 (cost=0.00..18.50 rows=349 width=12) (actual rows=350.00 loops=1) Filter: (r < '0.35'::double precision) Rows Removed by Filter: 650 -> Hash (cost=18.50..18.50 rows=389 width=12) (actual rows=391.00 loops=1) Buckets: 1024 Batches: 1 Memory Usage: 25kB - Bloom Filter 3: bits=8388608 hashes=10 memory=1024kB checked=10188 rejected=6191 + Bloom Filter 3: bits=8192 hashes=4 memory=1kB checked=10215 rejected=6199 -> Seq Scan on bloom_star_multi_dim_3 d3 (cost=0.00..18.50 rows=389 width=12) (actual rows=391.00 loops=1) Filter: (r < '0.4'::double precision) Rows Removed by Filter: 609 -> Hash (cost=18.50..18.50 rows=448 width=12) (actual rows=449.00 loops=1) Buckets: 1024 Batches: 1 Memory Usage: 28kB - Bloom Filter 4: bits=8388608 hashes=10 memory=1024kB checked=3997 rejected=2223 + Bloom Filter 4: bits=8192 hashes=4 memory=1kB checked=4016 rejected=2234 -> Seq Scan on bloom_star_multi_dim_4 d4 (cost=0.00..18.50 rows=448 width=12) (actual rows=449.00 loops=1) Filter: (r < '0.45'::double precision) Rows Removed by Filter: 551 -> Hash (cost=18.50..18.50 rows=518 width=12) (actual rows=519.00 loops=1) Buckets: 1024 Batches: 1 Memory Usage: 31kB - Bloom Filter 5: bits=8388608 hashes=10 memory=1024kB checked=1774 rejected=851 + Bloom Filter 5: bits=8192 hashes=4 memory=1kB checked=1782 rejected=848 -> Seq Scan on bloom_star_multi_dim_5 d5 (cost=0.00..18.50 rows=518 width=12) (actual rows=519.00 loops=1) Filter: (r < '0.5'::double precision) Rows Removed by Filter: 481 -> Hash (cost=18.50..18.50 rows=537 width=12) (actual rows=538.00 loops=1) Buckets: 1024 Batches: 1 Memory Usage: 32kB - Bloom Filter 6: bits=8388608 hashes=10 memory=1024kB checked=923 rejected=397 + Bloom Filter 6: bits=8192 hashes=4 memory=1kB checked=934 rejected=399 -> Seq Scan on bloom_star_multi_dim_6 d6 (cost=0.00..18.50 rows=537 width=12) (actual rows=538.00 loops=1) Filter: (r < '0.55'::double precision) Rows Removed by Filter: 462 -> Hash (cost=18.50..18.50 rows=563 width=12) (actual rows=564.00 loops=1) Buckets: 1024 Batches: 1 Memory Usage: 33kB - Bloom Filter 7: bits=8388608 hashes=10 memory=1024kB checked=526 rejected=234 + Bloom Filter 7: bits=8192 hashes=4 memory=1kB checked=535 rejected=236 -> Seq Scan on bloom_star_multi_dim_7 d7 (cost=0.00..18.50 rows=563 width=12) (actual rows=564.00 loops=1) Filter: (r < '0.6'::double precision) Rows Removed by Filter: 436 diff --git a/src/test/regress/expected/join.out b/src/test/regress/expected/join.out index 23c8b7a3c4a..7023b13763c 100644 --- a/src/test/regress/expected/join.out +++ b/src/test/regress/expected/join.out @@ -9583,20 +9583,20 @@ select * from fkest f1 join fkest f2 on (f1.x = f2.x and f1.x10 = f2.x10b and f1.x100 = f2.x100) join fkest f3 on f1.x = f3.x where f1.x100 = 2; - QUERY PLAN ------------------------------------------------------ + QUERY PLAN +----------------------------------------------------------------- Hash Join - Hash Cond: ((f1.x = f2.x) AND (f1.x10 = f2.x10b)) - -> Seq Scan on fkest f1 - Filter: (x100 = 2) + Hash Cond: (f3.x = f1.x) + -> Seq Scan on fkest f3 + Bloom Filter 1: keys=(x) -> Hash + Bloom Filter 1 -> Hash Join - Hash Cond: (f3.x = f2.x) - -> Seq Scan on fkest f3 - Bloom Filter 1: keys=(x) + Hash Cond: ((f2.x = f1.x) AND (f2.x10b = f1.x10)) + -> Seq Scan on fkest f2 + Filter: (x100 = 2) -> Hash - Bloom Filter 1 - -> Seq Scan on fkest f2 + -> Seq Scan on fkest f1 Filter: (x100 = 2) (13 rows) @@ -9786,52 +9786,44 @@ select * from j1 natural join j2; explain (verbose, costs off) select * from j1 inner join (select distinct id from j3) j3 on j1.id = j3.id; - QUERY PLAN ------------------------------------------------------ - Hash Join + QUERY PLAN +----------------------------------------- + Nested Loop Output: j1.id, j3.id Inner Unique: true - Hash Cond: (j1.id = j3.id) - -> Seq Scan on public.j1 - Output: j1.id - Bloom Filter 1: keys=(j1.id) expected=33.3% - -> Hash + Join Filter: (j1.id = j3.id) + -> Unique Output: j3.id - Bloom Filter 1 - -> Unique + -> Sort Output: j3.id - -> Sort + Sort Key: j3.id + -> Seq Scan on public.j3 Output: j3.id - Sort Key: j3.id - -> Seq Scan on public.j3 - Output: j3.id -(17 rows) + -> Seq Scan on public.j1 + Output: j1.id +(13 rows) -- ensure group by clause allows the inner to become unique explain (verbose, costs off) select * from j1 inner join (select id from j3 group by id) j3 on j1.id = j3.id; - QUERY PLAN ------------------------------------------------------ - Hash Join + QUERY PLAN +----------------------------------------- + Nested Loop Output: j1.id, j3.id Inner Unique: true - Hash Cond: (j1.id = j3.id) - -> Seq Scan on public.j1 - Output: j1.id - Bloom Filter 1: keys=(j1.id) expected=33.3% - -> Hash + Join Filter: (j1.id = j3.id) + -> Group Output: j3.id - Bloom Filter 1 - -> Group + Group Key: j3.id + -> Sort Output: j3.id - Group Key: j3.id - -> Sort + Sort Key: j3.id + -> Seq Scan on public.j3 Output: j3.id - Sort Key: j3.id - -> Seq Scan on public.j3 - Output: j3.id -(18 rows) + -> Seq Scan on public.j1 + Output: j1.id +(14 rows) drop table j1; drop table j2; diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out index 7fff6a720aa..b52528870ef 100644 --- a/src/test/regress/expected/misc_functions.out +++ b/src/test/regress/expected/misc_functions.out @@ -614,16 +614,14 @@ CREATE FUNCTION my_gen_series(int, int) RETURNS SETOF integer SUPPORT test_support_func; EXPLAIN (COSTS OFF) SELECT * FROM tenk1 a JOIN my_gen_series(1,1000) g ON a.unique1 = g; - QUERY PLAN ----------------------------------------------- + QUERY PLAN +---------------------------------------- Hash Join - Hash Cond: (a.unique1 = g.g) - -> Seq Scan on tenk1 a - Bloom Filter 1: keys=(unique1) + Hash Cond: (g.g = a.unique1) + -> Function Scan on my_gen_series g -> Hash - Bloom Filter 1 - -> Function Scan on my_gen_series g -(7 rows) + -> Seq Scan on tenk1 a +(5 rows) EXPLAIN (COSTS OFF) SELECT * FROM tenk1 a JOIN my_gen_series(1,10) g ON a.unique1 = g; diff --git a/src/test/regress/expected/returning.out b/src/test/regress/expected/returning.out index 7cfaa114137..53453326365 100644 --- a/src/test/regress/expected/returning.out +++ b/src/test/regress/expected/returning.out @@ -716,7 +716,7 @@ UPDATE joinview SET f3 = f3 + 1 WHERE f3 = 57 Hash Cond: (joinme_1.f2j = foo_1.f2) -> Seq Scan on pg_temp.joinme joinme_1 Output: joinme_1.ctid, joinme_1.f2j - Bloom Filter 2: keys=(joinme_1.f2j) expected=2.0% + Bloom Filter 2: keys=(joinme_1.f2j) expected=0.5% -> Hash Output: foo_1.f2, foo_1.tableoid, foo_1.ctid, joinme.ctid, joinme.other, joinme.f2j, foo_2.f1, foo_2.f3, foo_2.ctid, foo_2.f2, foo_2.tableoid Bloom Filter 2 @@ -725,7 +725,7 @@ UPDATE joinview SET f3 = f3 + 1 WHERE f3 = 57 Hash Cond: (joinme.f2j = foo_1.f2) -> Seq Scan on pg_temp.joinme Output: joinme.ctid, joinme.other, joinme.f2j - Bloom Filter 1: keys=(joinme.f2j) expected=2.0% + Bloom Filter 1: keys=(joinme.f2j) expected=0.5% -> Hash Output: foo_1.f2, foo_1.tableoid, foo_1.ctid, foo_2.f1, foo_2.f3, foo_2.ctid, foo_2.f2, foo_2.tableoid Bloom Filter 1 diff --git a/src/test/regress/expected/select_parallel.out b/src/test/regress/expected/select_parallel.out index 9d6d213d3e1..933921d1860 100644 --- a/src/test/regress/expected/select_parallel.out +++ b/src/test/regress/expected/select_parallel.out @@ -1125,27 +1125,28 @@ reset role; explain (costs off, verbose) select count(*) from tenk1 a where (unique1, two) in (select unique1, row_number() over() from tenk1 b); - QUERY PLAN ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + QUERY PLAN +---------------------------------------------------------------------------------------- Aggregate Output: count(*) - -> Hash Semi Join - Hash Cond: ((a.unique1 = b.unique1) AND (a.two = (row_number() OVER w1))) - -> Seq Scan on public.tenk1 a - Output: a.unique1, a.unique2, a.two, a.four, a.ten, a.twenty, a.hundred, a.thousand, a.twothousand, a.fivethous, a.tenthous, a.odd, a.even, a.stringu1, a.stringu2, a.string4 - Bloom Filter 1: keys=(a.unique1, a.two) expected=0.5% - -> Hash - Output: b.unique1, (row_number() OVER w1) - Bloom Filter 1 - -> WindowAgg - Output: b.unique1, row_number() OVER w1 - Window: w1 AS (ROWS UNBOUNDED PRECEDING) - -> Gather + -> Hash Right Semi Join + Hash Cond: ((b.unique1 = a.unique1) AND ((row_number() OVER w1) = a.two)) + -> WindowAgg + Output: b.unique1, row_number() OVER w1 + Window: w1 AS (ROWS UNBOUNDED PRECEDING) + -> Gather + Output: b.unique1 + Workers Planned: 4 + -> Parallel Index Only Scan using tenk1_unique1 on public.tenk1 b Output: b.unique1 - Workers Planned: 4 - -> Parallel Index Only Scan using tenk1_unique1 on public.tenk1 b - Output: b.unique1 -(18 rows) + -> Hash + Output: a.unique1, a.two + -> Gather + Output: a.unique1, a.two + Workers Planned: 4 + -> Parallel Seq Scan on public.tenk1 a + Output: a.unique1, a.two +(19 rows) -- LIMIT/OFFSET within sub-selects can't be pushed to workers. explain (costs off) -- 2.50.1 (Apple Git-155)