From e442d052a21260cefc0b7e48b6d4f7701dda5ed9 Mon Sep 17 00:00:00 2001 From: Tomas Vondra Date: Sat, 20 Jun 2026 20:44:00 +0200 Subject: [PATCH v6 1/9] PoC: hashjoin bloom filter pushdown When construction hashjoin plans, try to pushdown a Bloom filter built on the hashtable to a scan node in the outer side of the join. This has multiple significant benefits: a) Probing a bloom filter is cheaper than probing a hash table, so if a tuple gets eliminated using the bloom filter, we save cycles. b) The Bloom filter is more compact, and so more cache efficient. The hash table may not even split into memory, and the hashjoin has to spill data to files. The Bloom filter may still fit into memory, and eliminate many tuples on the outer side (which reduces the amount of data spilled to disk). c) The Bloom filter is pushed to a scan node, which may be multiple steps before the join. This increases the benefit, because the eliminated tuples don't need to pass through any of the nodes in between. This futehr amplifies the difference between probing a filter and probing the hashtable. d) If a table joins with multiple other tables (e.g. in a starjoin), the scan node may receive multiple Bloom filters. The selectivity of the filters multiply, once again amplifying the benefits. If a scan gets two filters, each discarding 90% of tuples, the scan will discard 99% of tuples, i.e. 2 orders of magnitude fewer tuples. We could try pushing down filters after the path construction, while the "createplan" builds the selected plan. That is relatively simple, but cardinalities and costs are already fixed at that point, and can't reflect the pushed down filter. That would make EXPLAIN rather confusing, difficult to decide if the difference is due to a filter or a genuine misestimate. It also means the filter can't affect the plan shape, because that's selected during the bottom-up phase. It can only improve the selected plan - but only if it happens to contain suitable hash joins. There may be a cheaper plan with pushed-down filters, but we won't generate it. Therefore, we consider filters during the bottom-up path construction, similarly to other path features, so that we can calculate correct cardinality estimates and costs. Each path gest a list of "expected filters", recording what filters the path requires during joins. This is a similar concept to path keys, tracking ordering of a path. When building scan paths for a bare relation, we figure out a list of "interesting" filters by inspecting the joins the relation participates in, and pick a limited number of sufficiently selective filters, and generate new scan paths for combinations of these filters. We need to be careful to keep the number of new paths under control. The number of combinations grows with 2^n, so with 3 interesting filters we get 9 paths (8 new ones), with 4 it's 16, etc. And this is for each scan node. For a query with many joins, we may also consider combinations of these new paths, etc. It can grow pretty quick. So we need to be conservative, in order to prevent an explosion of the number of paths. The patch simply picks a limited number of the most selective filters (e.g. 3 filters, each eliminating at least 30% tuples). We could refine how this is decided, of course. For example, CustomScan could/should have a say in the costing, somehow. When planning joins, we need to consider how the join interacts with filters expected by the input paths. * A join "matches" a filter of an input path, if it's the join from which the filter was derived. * A join can "satisfy" a filter expected by an outer input path, if it matches the filter, and if it's a hash join. A join can never satisfy filters expected by the inner input. * A join has to satisfy all filters matching the inner/outer path. This implies that: * HashJoins can satisfy filters expected by outer input path. * NestLoop/MergeJoin can never satisfy any filters. The places creating these joins have to ignore all paths with such "conflicting" filters (matching the join). * Any joins can propagate filters non-matching the join, so that a later joins may either satisfy them or ignore them. At execution time, the Hash node builds the filter with the hashtable, and the scan node probes the Bloom filter similarly to evaluating the regular quals. The effectivity of a filter is depends on how many tuples it discards. A highly selective filter (e.g. discarding >90% tuples) is going to be a win no matter what. But even a "poor" filter (e.g. discarding only 10% tuples) may still be a win, if the join has to spill data to disk. Accurately estimating the selectivity of a filter is hard, just like estimating join cardinality (because it's the same task, pretty much). We may underestimate the selectivity, and then the filter turns out to be useless (not discarding any rows). To mitigate the impact, the patch has a simple adaptive logic, to disable such useless filters, and then enable them again, when the filter gets more selective. Notes: * We could also consider the "remaining row count" after a filter, and only consider filters if it's above some limit. E.g. it there's less than 1000 tuples, there's little point in pushing down additional filters. This would help with reducing the impact of path explosion. * When figuring out interesting filters, we need to be careful about outer joins. Outer joins are not a good filter source, because we can't eliminate any rows even if the join clause is very selective. * We still do the pushdown when creating the plan, but it's very mechanical - all the decisions have been made earlier. It's not possible (allowed) to mismatch - we can't pushdown a filter not expected by a scan, and the scan can't expect a filter that has not been pushdown by the plan. * We have little control over the Bloom filter parameters. The library picks most of the attributes on our behalf. Works reasonably well, but we may need to know e.g. false positive rate (if it gets too high, the filter becomes useless, and we should stop using it). * We only push filters to scan nodes, and only through some other nodes (e.g. through joins, sort, ...). Maybe we could expand this to also pushdown through aggregations, etc. * The patch currently does not support parallel queries. This can be addressed later, it's certainly doable. But there are different cases. The filter may be built by a serial or parallel hash join, etc. * Similarly, there's no support for partitioned tables. Fixing this should be simpler than supporting parallel queries, I think. We'd just have to allow pushing to multiple consumers (one per partition). * It might be interesting to allow the scan nodes to use the Bloom filters in other ways. E.g. it might push the filter to storage, or perhaps to remote node (with a ForeignScan), and let it do smart things with it. The storage might prefilter data, foreign server could filter data on the remote end. That'd require using some well defined and portable library for the filter. * The cost model determining which filters are effective is a bit crude and based on empirical observations. For example the thresholds used in the adaptive logic are somewhat arbitrary and need more thought. * We could push filters into other nodes, not just scans. Might be useful for more complex joins. * We could also allow other filter types, not just Bloom. Like minmax range filters, IN() lists, etc. Or whatever the scan node wants. --- contrib/pg_plan_advice/expected/gather.out | 39 +- .../pg_plan_advice/expected/join_order.out | 68 ++- .../pg_plan_advice/expected/join_strategy.out | 47 +- contrib/pg_plan_advice/expected/semijoin.out | 70 ++- .../expected/pg_stash_advice.out | 144 +++-- .../postgres_fdw/expected/eval_plan_qual.out | 4 +- .../postgres_fdw/expected/postgres_fdw.out | 180 +++--- src/backend/commands/explain.c | 216 +++++++ src/backend/executor/execUtils.c | 2 + src/backend/executor/nodeBitmapHeapscan.c | 3 + src/backend/executor/nodeHash.c | 60 ++ src/backend/executor/nodeHashjoin.c | 456 +++++++++++++++ src/backend/executor/nodeIndexonlyscan.c | 3 + src/backend/executor/nodeIndexscan.c | 3 + src/backend/executor/nodeSamplescan.c | 3 + src/backend/executor/nodeSeqscan.c | 3 + src/backend/executor/nodeTidrangescan.c | 3 + src/backend/executor/nodeTidscan.c | 3 + src/backend/lib/bloomfilter.c | 19 + src/backend/optimizer/path/allpaths.c | 470 ++++++++++++++++ src/backend/optimizer/path/costsize.c | 18 + src/backend/optimizer/path/equivclass.c | 179 ++++++ src/backend/optimizer/path/joinpath.c | 526 +++++++++++++++--- src/backend/optimizer/plan/createplan.c | 234 ++++++++ src/backend/optimizer/plan/planner.c | 1 + src/backend/optimizer/plan/setrefs.c | 63 +++ src/backend/optimizer/util/pathnode.c | 158 ++++++ src/backend/utils/misc/guc_parameters.dat | 27 + src/backend/utils/misc/postgresql.conf.sample | 3 + src/include/executor/execScan.h | 22 +- src/include/executor/executor.h | 12 + src/include/executor/nodeHashjoin.h | 9 + src/include/lib/bloomfilter.h | 2 + src/include/nodes/execnodes.h | 91 +++ src/include/nodes/pathnodes.h | 69 +++ src/include/nodes/plannodes.h | 52 ++ src/include/optimizer/cost.h | 3 + src/include/optimizer/pathnode.h | 5 + src/include/optimizer/paths.h | 6 + src/test/regress/expected/aggregates.out | 8 +- src/test/regress/expected/eager_aggregate.out | 72 ++- src/test/regress/expected/graph_table.out | 14 +- src/test/regress/expected/join.out | 311 ++++++----- src/test/regress/expected/join_hash.out | 12 +- src/test/regress/expected/merge.out | 126 +++-- src/test/regress/expected/misc_functions.out | 18 +- src/test/regress/expected/privileges.out | 4 +- src/test/regress/expected/returning.out | 42 +- src/test/regress/expected/rowsecurity.out | 2 + src/test/regress/expected/select_views.out | 2 + src/test/regress/expected/stats_ext.out | 19 +- src/test/regress/expected/subselect.out | 35 +- src/test/regress/expected/sysviews.out | 3 +- src/test/regress/expected/updatable_views.out | 23 +- src/test/regress/expected/window.out | 19 +- src/test/regress/expected/with.out | 96 ++-- src/test/regress/sql/rowsecurity.sql | 3 + src/test/regress/sql/select_views.sql | 3 + src/tools/pgindent/typedefs.list | 2 + 59 files changed, 3418 insertions(+), 672 deletions(-) diff --git a/contrib/pg_plan_advice/expected/gather.out b/contrib/pg_plan_advice/expected/gather.out index 0cc0dedf859..6486d7c449d 100644 --- a/contrib/pg_plan_advice/expected/gather.out +++ b/contrib/pg_plan_advice/expected/gather.out @@ -18,23 +18,24 @@ VACUUM ANALYZE gt_fact; -- By default, we expect Gather Merge with a parallel hash join. EXPLAIN (COSTS OFF, PLAN_ADVICE) SELECT * FROM gt_fact f JOIN gt_dim d ON f.dim_id = d.id ORDER BY d.id; - QUERY PLAN -------------------------------------------------------- + QUERY PLAN +-------------------------------------------------- Gather Merge Workers Planned: 1 - -> Sort - Sort Key: f.dim_id - -> Parallel Hash Join - Hash Cond: (f.dim_id = d.id) + -> Merge Join + Merge Cond: (f.dim_id = d.id) + -> Sort + Sort Key: f.dim_id -> Parallel Seq Scan on gt_fact f - -> Parallel Hash - -> Parallel Seq Scan on gt_dim d + -> Sort + Sort Key: d.id + -> Seq Scan on gt_dim d Generated Plan Advice: JOIN_ORDER(f d) - HASH_JOIN(d) + MERGE_JOIN_PLAIN(d) SEQ_SCAN(f d) GATHER_MERGE((f d)) -(14 rows) +(15 rows) -- Force Gather or Gather Merge of both relations together. BEGIN; @@ -153,17 +154,18 @@ EXPLAIN (COSTS OFF, PLAN_ADVICE) -> Sort Sort Key: f.dim_id -> Parallel Seq Scan on gt_fact f - -> Index Scan using gt_dim_pkey on gt_dim d + -> Sort + Sort Key: d.id + -> Seq Scan on gt_dim d Supplied Plan Advice: GATHER((d d/d.d)) /* partially matched */ Generated Plan Advice: JOIN_ORDER(f d) MERGE_JOIN_PLAIN(d) - SEQ_SCAN(f) - INDEX_SCAN(d public.gt_dim_pkey) + SEQ_SCAN(f d) GATHER_MERGE(f) NO_GATHER(d) -(17 rows) +(18 rows) COMMIT; -- Force a Gather or Gather Merge on one relation but no parallelism on other. @@ -180,18 +182,19 @@ EXPLAIN (COSTS OFF, PLAN_ADVICE) -> Sort Sort Key: f.dim_id -> Parallel Seq Scan on gt_fact f - -> Index Scan using gt_dim_pkey on gt_dim d + -> Sort + Sort Key: d.id + -> Seq Scan on gt_dim d Supplied Plan Advice: GATHER_MERGE(f) /* matched */ NO_GATHER(d) /* matched */ Generated Plan Advice: JOIN_ORDER(f d) MERGE_JOIN_PLAIN(d) - SEQ_SCAN(f) - INDEX_SCAN(d public.gt_dim_pkey) + SEQ_SCAN(f d) GATHER_MERGE(f) NO_GATHER(d) -(18 rows) +(19 rows) SET LOCAL pg_plan_advice.advice = 'gather_merge(d) no_gather(f)'; EXPLAIN (COSTS OFF, PLAN_ADVICE) diff --git a/contrib/pg_plan_advice/expected/join_order.out b/contrib/pg_plan_advice/expected/join_order.out index a5a9728e3fd..048d725e548 100644 --- a/contrib/pg_plan_advice/expected/join_order.out +++ b/contrib/pg_plan_advice/expected/join_order.out @@ -27,25 +27,29 @@ SELECT * FROM jo_fact f LEFT JOIN jo_dim1 d1 ON f.dim1_id = d1.id LEFT JOIN jo_dim2 d2 ON f.dim2_id = d2.id WHERE val1 = 1 AND val2 = 1; - QUERY PLAN ------------------------------------------- + QUERY PLAN +---------------------------------------------- Hash Join - Hash Cond: (f.dim1_id = d1.id) + Hash Cond: (f.dim2_id = d2.id) -> Hash Join - Hash Cond: (f.dim2_id = d2.id) + Hash Cond: (f.dim1_id = d1.id) -> Seq Scan on jo_fact f + Bloom Filter 1: keys=(dim1_id) + Bloom Filter 2: keys=(dim2_id) -> Hash - -> Seq Scan on jo_dim2 d2 - Filter: (val2 = 1) + Bloom Filter 1 + -> Seq Scan on jo_dim1 d1 + Filter: (val1 = 1) -> Hash - -> Seq Scan on jo_dim1 d1 - Filter: (val1 = 1) + Bloom Filter 2 + -> Seq Scan on jo_dim2 d2 + Filter: (val2 = 1) Generated Plan Advice: - JOIN_ORDER(f d2 d1) - HASH_JOIN(d2 d1) - SEQ_SCAN(f d2 d1) + JOIN_ORDER(f d1 d2) + HASH_JOIN(d1 d2) + SEQ_SCAN(f d1 d2) NO_GATHER(f d1 d2) -(16 rows) +(20 rows) -- Force a few different join orders. Some of these are very inefficient, -- but the planner considers them all viable. @@ -56,17 +60,21 @@ SELECT * FROM jo_fact f LEFT JOIN jo_dim1 d1 ON f.dim1_id = d1.id LEFT JOIN jo_dim2 d2 ON f.dim2_id = d2.id WHERE val1 = 1 AND val2 = 1; - QUERY PLAN ------------------------------------------- + QUERY PLAN +---------------------------------------------- Hash Join Hash Cond: (f.dim2_id = d2.id) -> Hash Join Hash Cond: (f.dim1_id = d1.id) -> Seq Scan on jo_fact f + Bloom Filter 1: keys=(dim1_id) + Bloom Filter 2: keys=(dim2_id) -> Hash + Bloom Filter 1 -> Seq Scan on jo_dim1 d1 Filter: (val1 = 1) -> Hash + Bloom Filter 2 -> Seq Scan on jo_dim2 d2 Filter: (val2 = 1) Supplied Plan Advice: @@ -76,7 +84,7 @@ SELECT * FROM jo_fact f HASH_JOIN(d1 d2) SEQ_SCAN(f d1 d2) NO_GATHER(f d1 d2) -(18 rows) +(22 rows) SET LOCAL pg_plan_advice.advice = 'join_order(f d2 d1)'; EXPLAIN (COSTS OFF, PLAN_ADVICE) @@ -84,17 +92,21 @@ SELECT * FROM jo_fact f LEFT JOIN jo_dim1 d1 ON f.dim1_id = d1.id LEFT JOIN jo_dim2 d2 ON f.dim2_id = d2.id WHERE val1 = 1 AND val2 = 1; - QUERY PLAN ------------------------------------------- + QUERY PLAN +---------------------------------------------- Hash Join Hash Cond: (f.dim1_id = d1.id) -> Hash Join Hash Cond: (f.dim2_id = d2.id) -> Seq Scan on jo_fact f + Bloom Filter 1: keys=(dim2_id) + Bloom Filter 2: keys=(dim1_id) -> Hash + Bloom Filter 1 -> Seq Scan on jo_dim2 d2 Filter: (val2 = 1) -> Hash + Bloom Filter 2 -> Seq Scan on jo_dim1 d1 Filter: (val1 = 1) Supplied Plan Advice: @@ -104,7 +116,7 @@ SELECT * FROM jo_fact f HASH_JOIN(d2 d1) SEQ_SCAN(f d2 d1) NO_GATHER(f d1 d2) -(18 rows) +(22 rows) SET LOCAL pg_plan_advice.advice = 'join_order(d1 f d2)'; EXPLAIN (COSTS OFF, PLAN_ADVICE) @@ -145,7 +157,9 @@ SELECT * FROM jo_fact f Hash Join Hash Cond: ((f.dim1_id = d1.id) AND (f.dim2_id = d2.id)) -> Seq Scan on jo_fact f + Bloom Filter 1: keys=(dim1_id, dim2_id) -> Hash + Bloom Filter 1 -> Nested Loop -> Seq Scan on jo_dim1 d1 Filter: (val1 = 1) @@ -160,7 +174,7 @@ SELECT * FROM jo_fact f HASH_JOIN((d1 d2)) SEQ_SCAN(f d1 d2) NO_GATHER(f d1 d2) -(18 rows) +(20 rows) SET LOCAL pg_plan_advice.advice = 'join_order(f {d1 d2})'; EXPLAIN (COSTS OFF, PLAN_ADVICE) @@ -173,7 +187,9 @@ SELECT * FROM jo_fact f Hash Join Hash Cond: ((f.dim1_id = d1.id) AND (f.dim2_id = d2.id)) -> Seq Scan on jo_fact f + Bloom Filter 1: keys=(dim1_id, dim2_id) -> Hash + Bloom Filter 1 -> Nested Loop -> Seq Scan on jo_dim1 d1 Filter: (val1 = 1) @@ -188,7 +204,7 @@ SELECT * FROM jo_fact f HASH_JOIN((d1 d2)) SEQ_SCAN(f d1 d2) NO_GATHER(f d1 d2) -(18 rows) +(20 rows) COMMIT; -- Force a join order by mentioning just a prefix of the join list. @@ -199,17 +215,21 @@ SELECT * FROM jo_fact f LEFT JOIN jo_dim1 d1 ON f.dim1_id = d1.id LEFT JOIN jo_dim2 d2 ON f.dim2_id = d2.id WHERE val1 = 1 AND val2 = 1; - QUERY PLAN ------------------------------------------------- + QUERY PLAN +---------------------------------------------------- Hash Join Hash Cond: (d2.id = f.dim2_id) -> Seq Scan on jo_dim2 d2 Filter: (val2 = 1) + Bloom Filter 2: keys=(id) -> Hash + Bloom Filter 2 -> Hash Join Hash Cond: (f.dim1_id = d1.id) -> Seq Scan on jo_fact f + Bloom Filter 1: keys=(dim1_id) -> Hash + Bloom Filter 1 -> Seq Scan on jo_dim1 d1 Filter: (val1 = 1) Supplied Plan Advice: @@ -219,7 +239,7 @@ SELECT * FROM jo_fact f HASH_JOIN(d1 (f d1)) SEQ_SCAN(d2 f d1) NO_GATHER(f d1 d2) -(18 rows) +(22 rows) SET LOCAL pg_plan_advice.advice = 'join_order(d2 d1)'; EXPLAIN (COSTS OFF, PLAN_ADVICE) @@ -292,7 +312,7 @@ SELECT * FROM jo_fact f -------------------------------------------------------------- Nested Loop Disabled: true - Join Filter: ((d1.id = f.dim1_id) AND (d2.id = f.dim2_id)) + Join Filter: ((f.dim1_id = d1.id) AND (f.dim2_id = d2.id)) -> Nested Loop -> Seq Scan on jo_dim1 d1 Filter: (val1 = 1) diff --git a/contrib/pg_plan_advice/expected/join_strategy.out b/contrib/pg_plan_advice/expected/join_strategy.out index 0f9db692190..fad68bfd18b 100644 --- a/contrib/pg_plan_advice/expected/join_strategy.out +++ b/contrib/pg_plan_advice/expected/join_strategy.out @@ -56,39 +56,45 @@ EXPLAIN (COSTS OFF, PLAN_ADVICE) SET LOCAL pg_plan_advice.advice = 'MERGE_JOIN_MATERIALIZE(d)'; EXPLAIN (COSTS OFF, PLAN_ADVICE) SELECT * FROM join_fact f JOIN join_dim d ON f.dim_id = d.id; - QUERY PLAN ----------------------------------------------------------------- + QUERY PLAN +-------------------------------------------------------- Merge Join Disabled: true Merge Cond: (f.dim_id = d.id) -> Index Scan using join_fact_dim_id on join_fact f - -> Index Scan using join_dim_pkey on join_dim d + -> Sort + Sort Key: d.id + -> Seq Scan on join_dim d Supplied Plan Advice: MERGE_JOIN_MATERIALIZE(d) /* matched, failed */ Generated Plan Advice: JOIN_ORDER(f d) MERGE_JOIN_PLAIN(d) - INDEX_SCAN(f public.join_fact_dim_id d public.join_dim_pkey) + SEQ_SCAN(d) + INDEX_SCAN(f public.join_fact_dim_id) NO_GATHER(f d) -(12 rows) +(15 rows) SET LOCAL pg_plan_advice.advice = 'MERGE_JOIN_PLAIN(d)'; EXPLAIN (COSTS OFF, PLAN_ADVICE) SELECT * FROM join_fact f JOIN join_dim d ON f.dim_id = d.id; - QUERY PLAN ----------------------------------------------------------------- + QUERY PLAN +-------------------------------------------------------- Merge Join Merge Cond: (f.dim_id = d.id) -> Index Scan using join_fact_dim_id on join_fact f - -> Index Scan using join_dim_pkey on join_dim d + -> Sort + Sort Key: d.id + -> Seq Scan on join_dim d Supplied Plan Advice: MERGE_JOIN_PLAIN(d) /* matched */ Generated Plan Advice: JOIN_ORDER(f d) MERGE_JOIN_PLAIN(d) - INDEX_SCAN(f public.join_fact_dim_id d public.join_dim_pkey) + SEQ_SCAN(d) + INDEX_SCAN(f public.join_fact_dim_id) NO_GATHER(f d) -(11 rows) +(14 rows) SET LOCAL pg_plan_advice.advice = 'NESTED_LOOP_MATERIALIZE(d)'; EXPLAIN (COSTS OFF, PLAN_ADVICE) @@ -162,7 +168,9 @@ EXPLAIN (COSTS OFF, PLAN_ADVICE) Hash Join Hash Cond: (d.id = f.dim_id) -> Seq Scan on join_dim d + Bloom Filter 1: keys=(id) -> Hash + Bloom Filter 1 -> Seq Scan on join_fact f Supplied Plan Advice: HASH_JOIN(f) /* matched */ @@ -171,7 +179,7 @@ EXPLAIN (COSTS OFF, PLAN_ADVICE) HASH_JOIN(f) SEQ_SCAN(d f) NO_GATHER(f d) -(12 rows) +(14 rows) SET LOCAL pg_plan_advice.advice = 'MERGE_JOIN_MATERIALIZE(f)'; EXPLAIN (COSTS OFF, PLAN_ADVICE) @@ -301,19 +309,22 @@ EXPLAIN (COSTS OFF, PLAN_ADVICE) SELECT * FROM join_fact f JOIN join_dim d ON f.dim_id = d.id; QUERY PLAN ----------------------------------------------------------------- - Merge Join - Merge Cond: (d.id = f.dim_id) - -> Index Scan using join_dim_pkey on join_dim d - -> Index Scan using join_fact_dim_id on join_fact f + Hash Join + Hash Cond: (d.id = f.dim_id) + -> Seq Scan on join_dim d + Bloom Filter 1: keys=(id) + -> Hash + Bloom Filter 1 + -> Seq Scan on join_fact f Supplied Plan Advice: NESTED_LOOP_PLAIN(f) /* matched, conflicting, failed */ NESTED_LOOP_MATERIALIZE(f) /* matched, conflicting, failed */ Generated Plan Advice: JOIN_ORDER(d f) - MERGE_JOIN_PLAIN(f) - INDEX_SCAN(d public.join_dim_pkey f public.join_fact_dim_id) + HASH_JOIN(f) + SEQ_SCAN(d f) NO_GATHER(f d) -(12 rows) +(15 rows) SET LOCAL pg_plan_advice.advice = 'NESTED_LOOP_PLAIN(f d)'; EXPLAIN (COSTS OFF, PLAN_ADVICE) diff --git a/contrib/pg_plan_advice/expected/semijoin.out b/contrib/pg_plan_advice/expected/semijoin.out index db6b069ec8e..02c5e5ec9c7 100644 --- a/contrib/pg_plan_advice/expected/semijoin.out +++ b/contrib/pg_plan_advice/expected/semijoin.out @@ -75,7 +75,9 @@ SELECT * FROM sj_wide Hash Semi Join Hash Cond: ((sj_wide.id = "*VALUES*".column1) AND (sj_wide.val1 = "*VALUES*".column2)) -> Seq Scan on sj_wide + Bloom Filter 1: keys=(id, val1) -> Hash + Bloom Filter 1 -> Values Scan on "*VALUES*" Supplied Plan Advice: SEMIJOIN_NON_UNIQUE("*VALUES*") /* matched */ @@ -85,7 +87,7 @@ SELECT * FROM sj_wide SEQ_SCAN(sj_wide) SEMIJOIN_NON_UNIQUE("*VALUES*") NO_GATHER(sj_wide "*VALUES*") -(13 rows) +(15 rows) COMMIT; -- Because this table is narrower than the previous one, a sequential scan @@ -100,7 +102,9 @@ SELECT * FROM sj_narrow Hash Semi Join Hash Cond: ((sj_narrow.id = "*VALUES*".column1) AND (sj_narrow.val1 = "*VALUES*".column2)) -> Seq Scan on sj_narrow + Bloom Filter 1: keys=(id, val1) -> Hash + Bloom Filter 1 -> Values Scan on "*VALUES*" Generated Plan Advice: JOIN_ORDER(sj_narrow "*VALUES*") @@ -108,7 +112,7 @@ SELECT * FROM sj_narrow SEQ_SCAN(sj_narrow) SEMIJOIN_NON_UNIQUE("*VALUES*") NO_GATHER(sj_narrow "*VALUES*") -(11 rows) +(13 rows) -- Here, we expect advising a unique semijoin to swith to the same plan that -- we got with sj_wide, and advising a non-unique semijoin should not change @@ -123,7 +127,9 @@ SELECT * FROM sj_narrow Hash Join Hash Cond: ((sj_narrow.id = "*VALUES*".column1) AND (sj_narrow.val1 = "*VALUES*".column2)) -> Seq Scan on sj_narrow + Bloom Filter 1: keys=(id, val1) -> Hash + Bloom Filter 1 -> HashAggregate Group Key: "*VALUES*".column1, "*VALUES*".column2 -> Values Scan on "*VALUES*" @@ -135,7 +141,7 @@ SELECT * FROM sj_narrow SEQ_SCAN(sj_narrow) SEMIJOIN_UNIQUE("*VALUES*") NO_GATHER(sj_narrow "*VALUES*") -(15 rows) +(17 rows) SET LOCAL pg_plan_advice.advice = 'semijoin_non_unique("*VALUES*")'; EXPLAIN (COSTS OFF, PLAN_ADVICE) @@ -146,7 +152,9 @@ SELECT * FROM sj_narrow Hash Semi Join Hash Cond: ((sj_narrow.id = "*VALUES*".column1) AND (sj_narrow.val1 = "*VALUES*".column2)) -> Seq Scan on sj_narrow + Bloom Filter 1: keys=(id, val1) -> Hash + Bloom Filter 1 -> Values Scan on "*VALUES*" Supplied Plan Advice: SEMIJOIN_NON_UNIQUE("*VALUES*") /* matched */ @@ -156,7 +164,7 @@ SELECT * FROM sj_narrow SEQ_SCAN(sj_narrow) SEMIJOIN_NON_UNIQUE("*VALUES*") NO_GATHER(sj_narrow "*VALUES*") -(13 rows) +(15 rows) COMMIT; -- In the above example, we made the outer side of the join unique, but here, @@ -261,7 +269,9 @@ SELECT * FROM generate_series(1,1000) g Hash Right Semi Join Hash Cond: (sj_narrow.val1 = g.g) -> Seq Scan on sj_narrow + Bloom Filter 1: keys=(val1) -> Hash + Bloom Filter 1 -> Function Scan on generate_series g Supplied Plan Advice: SEMIJOIN_NON_UNIQUE(sj_narrow) /* matched */ @@ -272,7 +282,7 @@ SELECT * FROM generate_series(1,1000) g SEQ_SCAN(sj_narrow) SEMIJOIN_NON_UNIQUE(sj_narrow) NO_GATHER(g sj_narrow) -(14 rows) +(16 rows) COMMIT; -- However, mentioning the wrong side of the join should result in an advice @@ -359,20 +369,21 @@ EXPLAIN (COSTS OFF, PLAN_ADVICE) SELECT * FROM generate_series(1,1000) g, sj_narrow s WHERE g = s.val1; QUERY PLAN ---------------------------------------------------------- - Merge Join - Merge Cond: (s.val1 = g.g) - -> Index Scan using sj_narrow_val1_idx on sj_narrow s - -> Sort - Sort Key: g.g + Hash Join + Hash Cond: (s.val1 = g.g) + -> Seq Scan on sj_narrow s + Bloom Filter 1: keys=(val1) + -> Hash + Bloom Filter 1 -> Function Scan on generate_series g Supplied Plan Advice: SEMIJOIN_UNIQUE(g) /* matched, inapplicable, failed */ Generated Plan Advice: JOIN_ORDER(s g) - MERGE_JOIN_PLAIN(g) - INDEX_SCAN(s public.sj_narrow_val1_idx) + HASH_JOIN(g) + SEQ_SCAN(s) NO_GATHER(g s) -(13 rows) +(14 rows) COMMIT; -- Test the case where the subquery containing a semijoin is removed from @@ -400,27 +411,28 @@ SELECT 1 FROM generate_series(1, 1000) g WHERE EXISTS (SELECT 1 FROM (SELECT 1 FROM (SELECT 1) LEFT JOIN sj_narrow ON true) s, sj_narrow t2 WHERE g = t2.id); - QUERY PLAN ------------------------------------------------------------------------- - Hash Join - Hash Cond: (t2.id = g.g) + QUERY PLAN +------------------------------------------------------------------------------ + Merge Join + Merge Cond: (g.g = t2.id) + -> Sort + Sort Key: g.g + -> Function Scan on generate_series g -> Unique - -> Nested Loop - -> Index Only Scan using sj_narrow_pkey on sj_narrow t2 - -> Materialize - -> Nested Loop Left Join + -> Nested Loop Left Join + -> Nested Loop + -> Index Only Scan using sj_narrow_pkey on sj_narrow t2 + -> Materialize -> Result - -> Seq Scan on sj_narrow - -> Hash - -> Function Scan on generate_series g + -> Materialize + -> Seq Scan on sj_narrow Generated Plan Advice: - JOIN_ORDER(t2 ("*RESULT*" sj_narrow) g) - NESTED_LOOP_PLAIN(sj_narrow) - NESTED_LOOP_MATERIALIZE((sj_narrow "*RESULT*")) - HASH_JOIN(g) + JOIN_ORDER(g (t2 "*RESULT*" sj_narrow)) + MERGE_JOIN_PLAIN((t2 sj_narrow "*RESULT*")) + NESTED_LOOP_MATERIALIZE("*RESULT*" sj_narrow) SEQ_SCAN(sj_narrow) INDEX_ONLY_SCAN(t2 public.sj_narrow_pkey) SEMIJOIN_UNIQUE((t2 sj_narrow "*RESULT*")) NO_GATHER(g t2 sj_narrow "*RESULT*") -(20 rows) +(21 rows) diff --git a/contrib/pg_stash_advice/expected/pg_stash_advice.out b/contrib/pg_stash_advice/expected/pg_stash_advice.out index 788da854aa7..79e6e3642d5 100644 --- a/contrib/pg_stash_advice/expected/pg_stash_advice.out +++ b/contrib/pg_stash_advice/expected/pg_stash_advice.out @@ -57,20 +57,24 @@ EXPLAIN (COSTS OFF) SELECT * FROM aa_fact f LEFT JOIN aa_dim1 d1 ON f.dim1_id = d1.id LEFT JOIN aa_dim2 d2 ON f.dim2_id = d2.id WHERE val1 = 1 AND val2 = 1; - QUERY PLAN ------------------------------------------- + QUERY PLAN +---------------------------------------------- Hash Join - Hash Cond: (f.dim1_id = d1.id) + Hash Cond: (f.dim2_id = d2.id) -> Hash Join - Hash Cond: (f.dim2_id = d2.id) + Hash Cond: (f.dim1_id = d1.id) -> Seq Scan on aa_fact f + Bloom Filter 1: keys=(dim1_id) + Bloom Filter 2: keys=(dim2_id) -> Hash - -> Seq Scan on aa_dim2 d2 - Filter: (val2 = 1) + Bloom Filter 1 + -> Seq Scan on aa_dim1 d1 + Filter: (val1 = 1) -> Hash - -> Seq Scan on aa_dim1 d1 - Filter: (val1 = 1) -(11 rows) + Bloom Filter 2 + -> Seq Scan on aa_dim2 d2 + Filter: (val2 = 1) +(15 rows) -- Force an index scan on dim1 SELECT pg_set_stashed_advice('regress_stash', :'qid', @@ -84,22 +88,26 @@ EXPLAIN (COSTS OFF) SELECT * FROM aa_fact f LEFT JOIN aa_dim1 d1 ON f.dim1_id = d1.id LEFT JOIN aa_dim2 d2 ON f.dim2_id = d2.id WHERE val1 = 1 AND val2 = 1; - QUERY PLAN ---------------------------------------------------------- + QUERY PLAN +--------------------------------------------------------------- Hash Join - Hash Cond: (f.dim1_id = d1.id) + Hash Cond: (f.dim2_id = d2.id) -> Hash Join - Hash Cond: (f.dim2_id = d2.id) + Hash Cond: (f.dim1_id = d1.id) -> Seq Scan on aa_fact f + Bloom Filter 1: keys=(dim1_id) + Bloom Filter 2: keys=(dim2_id) -> Hash - -> Seq Scan on aa_dim2 d2 - Filter: (val2 = 1) + Bloom Filter 1 + -> Index Scan using aa_dim1_pkey on aa_dim1 d1 + Filter: (val1 = 1) -> Hash - -> Index Scan using aa_dim1_pkey on aa_dim1 d1 - Filter: (val1 = 1) + Bloom Filter 2 + -> Seq Scan on aa_dim2 d2 + Filter: (val2 = 1) Supplied Plan Advice: INDEX_SCAN(d1 aa_dim1_pkey) /* matched */ -(13 rows) +(17 rows) -- Force an alternative join order SELECT pg_set_stashed_advice('regress_stash', :'qid', @@ -113,22 +121,26 @@ EXPLAIN (COSTS OFF) SELECT * FROM aa_fact f LEFT JOIN aa_dim1 d1 ON f.dim1_id = d1.id LEFT JOIN aa_dim2 d2 ON f.dim2_id = d2.id WHERE val1 = 1 AND val2 = 1; - QUERY PLAN ------------------------------------------- + QUERY PLAN +---------------------------------------------- Hash Join Hash Cond: (f.dim2_id = d2.id) -> Hash Join Hash Cond: (f.dim1_id = d1.id) -> Seq Scan on aa_fact f + Bloom Filter 1: keys=(dim1_id) + Bloom Filter 2: keys=(dim2_id) -> Hash + Bloom Filter 1 -> Seq Scan on aa_dim1 d1 Filter: (val1 = 1) -> Hash + Bloom Filter 2 -> Seq Scan on aa_dim2 d2 Filter: (val2 = 1) Supplied Plan Advice: JOIN_ORDER(f d1 d2) /* matched */ -(13 rows) +(17 rows) -- Force an alternative join strategy SELECT pg_set_stashed_advice('regress_stash', :'qid', @@ -142,21 +154,22 @@ EXPLAIN (COSTS OFF) SELECT * FROM aa_fact f LEFT JOIN aa_dim1 d1 ON f.dim1_id = d1.id LEFT JOIN aa_dim2 d2 ON f.dim2_id = d2.id WHERE val1 = 1 AND val2 = 1; - QUERY PLAN ---------------------------------------------------- - Nested Loop - -> Hash Join - Hash Cond: (f.dim2_id = d2.id) - -> Seq Scan on aa_fact f - -> Hash + QUERY PLAN +------------------------------------------------------------ + Hash Join + Hash Cond: ((f.dim1_id = d1.id) AND (f.dim2_id = d2.id)) + -> Seq Scan on aa_fact f + Bloom Filter 1: keys=(dim1_id, dim2_id) + -> Hash + Bloom Filter 1 + -> Nested Loop -> Seq Scan on aa_dim2 d2 Filter: (val2 = 1) - -> Index Scan using aa_dim1_pkey on aa_dim1 d1 - Index Cond: (id = f.dim1_id) - Filter: (val1 = 1) + -> Seq Scan on aa_dim1 d1 + Filter: (val1 = 1) Supplied Plan Advice: NESTED_LOOP_PLAIN(d1) /* matched */ -(12 rows) +(13 rows) -- Add a useless extra entry to our test stash. Shouldn't change the result -- from the previous test. @@ -172,21 +185,22 @@ EXPLAIN (COSTS OFF) SELECT * FROM aa_fact f LEFT JOIN aa_dim1 d1 ON f.dim1_id = d1.id LEFT JOIN aa_dim2 d2 ON f.dim2_id = d2.id WHERE val1 = 1 AND val2 = 1; - QUERY PLAN ---------------------------------------------------- - Nested Loop - -> Hash Join - Hash Cond: (f.dim2_id = d2.id) - -> Seq Scan on aa_fact f - -> Hash + QUERY PLAN +------------------------------------------------------------ + Hash Join + Hash Cond: ((f.dim1_id = d1.id) AND (f.dim2_id = d2.id)) + -> Seq Scan on aa_fact f + Bloom Filter 1: keys=(dim1_id, dim2_id) + -> Hash + Bloom Filter 1 + -> Nested Loop -> Seq Scan on aa_dim2 d2 Filter: (val2 = 1) - -> Index Scan using aa_dim1_pkey on aa_dim1 d1 - Index Cond: (id = f.dim1_id) - Filter: (val1 = 1) + -> Seq Scan on aa_dim1 d1 + Filter: (val1 = 1) Supplied Plan Advice: NESTED_LOOP_PLAIN(d1) /* matched */ -(12 rows) +(13 rows) -- Try an empty stash to be sure it does nothing SELECT pg_create_advice_stash('regress_empty_stash'); @@ -200,20 +214,24 @@ EXPLAIN (COSTS OFF) SELECT * FROM aa_fact f LEFT JOIN aa_dim1 d1 ON f.dim1_id = d1.id LEFT JOIN aa_dim2 d2 ON f.dim2_id = d2.id WHERE val1 = 1 AND val2 = 1; - QUERY PLAN ------------------------------------------- + QUERY PLAN +---------------------------------------------- Hash Join - Hash Cond: (f.dim1_id = d1.id) + Hash Cond: (f.dim2_id = d2.id) -> Hash Join - Hash Cond: (f.dim2_id = d2.id) + Hash Cond: (f.dim1_id = d1.id) -> Seq Scan on aa_fact f + Bloom Filter 1: keys=(dim1_id) + Bloom Filter 2: keys=(dim2_id) -> Hash - -> Seq Scan on aa_dim2 d2 - Filter: (val2 = 1) + Bloom Filter 1 + -> Seq Scan on aa_dim1 d1 + Filter: (val1 = 1) -> Hash - -> Seq Scan on aa_dim1 d1 - Filter: (val1 = 1) -(11 rows) + Bloom Filter 2 + -> Seq Scan on aa_dim2 d2 + Filter: (val2 = 1) +(15 rows) -- Test that we can list each stash individually and all of them together, -- but not a nonexistent stash. @@ -263,20 +281,24 @@ EXPLAIN (COSTS OFF) SELECT * FROM aa_fact f LEFT JOIN aa_dim1 d1 ON f.dim1_id = d1.id LEFT JOIN aa_dim2 d2 ON f.dim2_id = d2.id WHERE val1 = 1 AND val2 = 1; - QUERY PLAN ------------------------------------------- + QUERY PLAN +---------------------------------------------- Hash Join - Hash Cond: (f.dim1_id = d1.id) + Hash Cond: (f.dim2_id = d2.id) -> Hash Join - Hash Cond: (f.dim2_id = d2.id) + Hash Cond: (f.dim1_id = d1.id) -> Seq Scan on aa_fact f + Bloom Filter 1: keys=(dim1_id) + Bloom Filter 2: keys=(dim2_id) -> Hash - -> Seq Scan on aa_dim2 d2 - Filter: (val2 = 1) + Bloom Filter 1 + -> Seq Scan on aa_dim1 d1 + Filter: (val1 = 1) -> Hash - -> Seq Scan on aa_dim1 d1 - Filter: (val1 = 1) -(11 rows) + Bloom Filter 2 + -> Seq Scan on aa_dim2 d2 + Filter: (val2 = 1) +(15 rows) SELECT * FROM pg_get_advice_stashes() ORDER BY stash_name; stash_name | num_entries diff --git a/contrib/postgres_fdw/expected/eval_plan_qual.out b/contrib/postgres_fdw/expected/eval_plan_qual.out index 5361fe6f329..b14d882e0ae 100644 --- a/contrib/postgres_fdw/expected/eval_plan_qual.out +++ b/contrib/postgres_fdw/expected/eval_plan_qual.out @@ -49,7 +49,7 @@ LockRows Filter: (l.v = 'foo'::text) -> Foreign Scan on public.ft Output: ft.*, ft.i - Remote SQL: SELECT i, v FROM public.t WHERE ((i = $1::integer)) + Remote SQL: SELECT i, v FROM public.t WHERE (($1::integer = i)) (10 rows) i|v @@ -73,7 +73,7 @@ LockRows Output: a.i, a.ctid, fb.*, fc.* -> Nested Loop Output: a.i, a.ctid, fb.*, fc.* - Join Filter: (fb.i = a.i) + Join Filter: (a.i = fb.i) -> Foreign Scan Output: fb.*, fb.i, fc.*, fc.i Relations: (public.fb) INNER JOIN (public.fc) diff --git a/contrib/postgres_fdw/expected/postgres_fdw.out b/contrib/postgres_fdw/expected/postgres_fdw.out index 9303de98b62..83bd96be3f3 100644 --- a/contrib/postgres_fdw/expected/postgres_fdw.out +++ b/contrib/postgres_fdw/expected/postgres_fdw.out @@ -477,20 +477,21 @@ SET enable_nestloop TO false; -- inner join; expressions in the clauses appear in the equivalence class list EXPLAIN (VERBOSE, COSTS OFF) SELECT t1.c1, t2."C 1" FROM ft2 t1 JOIN "S 1"."T 1" t2 ON (t1.c1 = t2."C 1") OFFSET 100 LIMIT 10; - QUERY PLAN ---------------------------------------------------------------------------------------- + QUERY PLAN +--------------------------------------------------------------------------------------------- Limit Output: t1.c1, t2."C 1" -> Merge Join Output: t1.c1, t2."C 1" - Inner Unique: true - Merge Cond: (t1.c1 = t2."C 1") - -> Foreign Scan on public.ft2 t1 - Output: t1.c1 - Remote SQL: SELECT "C 1" FROM "S 1"."T 1" ORDER BY "C 1" ASC NULLS LAST + Merge Cond: (t2."C 1" = t1.c1) -> Index Only Scan using t1_pkey on "S 1"."T 1" t2 Output: t2."C 1" -(11 rows) + -> Materialize + Output: t1.c1 + -> Foreign Scan on public.ft2 t1 + Output: t1.c1 + Remote SQL: SELECT "C 1" FROM "S 1"."T 1" ORDER BY "C 1" ASC NULLS LAST +(12 rows) SELECT t1.c1, t2."C 1" FROM ft2 t1 JOIN "S 1"."T 1" t2 ON (t1.c1 = t2."C 1") OFFSET 100 LIMIT 10; c1 | C 1 @@ -545,21 +546,22 @@ SELECT t1.c1, t2."C 1" FROM ft2 t1 LEFT JOIN "S 1"."T 1" t2 ON (t1.c1 = t2."C 1" -- foreign join so that the local table can be joined using merge join strategy. EXPLAIN (VERBOSE, COSTS OFF) SELECT t1."C 1" FROM "S 1"."T 1" t1 left join ft1 t2 join ft2 t3 on (t2.c1 = t3.c1) on (t3.c1 = t1."C 1") OFFSET 100 LIMIT 10; - QUERY PLAN ---------------------------------------------------------------------------------------------------------------------------------------------------------- + QUERY PLAN +--------------------------------------------------------------------------------------------------------------------------------------------------------------- Limit Output: t1."C 1" - -> Merge Right Join + -> Merge Left Join Output: t1."C 1" - Inner Unique: true - Merge Cond: (t3.c1 = t1."C 1") - -> Foreign Scan - Output: t3.c1 - Relations: (public.ft1 t2) INNER JOIN (public.ft2 t3) - Remote SQL: SELECT r3."C 1" FROM ("S 1"."T 1" r2 INNER JOIN "S 1"."T 1" r3 ON (((r3."C 1" = r2."C 1")))) ORDER BY r2."C 1" ASC NULLS LAST + Merge Cond: (t1."C 1" = t3.c1) -> Index Only Scan using t1_pkey on "S 1"."T 1" t1 Output: t1."C 1" -(12 rows) + -> Materialize + Output: t3.c1 + -> Foreign Scan + Output: t3.c1 + Relations: (public.ft1 t2) INNER JOIN (public.ft2 t3) + Remote SQL: SELECT r3."C 1" FROM ("S 1"."T 1" r2 INNER JOIN "S 1"."T 1" r3 ON (((r2."C 1" = r3."C 1")))) ORDER BY r2."C 1" ASC NULLS LAST +(13 rows) SELECT t1."C 1" FROM "S 1"."T 1" t1 left join ft1 t2 join ft2 t3 on (t2.c1 = t3.c1) on (t3.c1 = t1."C 1") OFFSET 100 LIMIT 10; C 1 @@ -581,21 +583,22 @@ SELECT t1."C 1" FROM "S 1"."T 1" t1 left join ft1 t2 join ft2 t3 on (t2.c1 = t3. -- included in join restrictions. EXPLAIN (VERBOSE, COSTS OFF) SELECT t1."C 1", t2.c1, t3.c1 FROM "S 1"."T 1" t1 left join ft1 t2 full join ft2 t3 on (t2.c1 = t3.c1) on (t3.c1 = t1."C 1") OFFSET 100 LIMIT 10; - QUERY PLAN ------------------------------------------------------------------------------------------------------------------------------------------------------------------- + QUERY PLAN +------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Limit Output: t1."C 1", t2.c1, t3.c1 - -> Merge Right Join + -> Merge Left Join Output: t1."C 1", t2.c1, t3.c1 - Inner Unique: true - Merge Cond: (t3.c1 = t1."C 1") - -> Foreign Scan - Output: t3.c1, t2.c1 - Relations: (public.ft2 t3) LEFT JOIN (public.ft1 t2) - Remote SQL: SELECT r3."C 1", r2."C 1" FROM ("S 1"."T 1" r3 LEFT JOIN "S 1"."T 1" r2 ON (((r2."C 1" = r3."C 1")))) ORDER BY r3."C 1" ASC NULLS LAST + Merge Cond: (t1."C 1" = t3.c1) -> Index Only Scan using t1_pkey on "S 1"."T 1" t1 Output: t1."C 1" -(12 rows) + -> Materialize + Output: t3.c1, t2.c1 + -> Foreign Scan + Output: t3.c1, t2.c1 + Relations: (public.ft2 t3) LEFT JOIN (public.ft1 t2) + Remote SQL: SELECT r3."C 1", r2."C 1" FROM ("S 1"."T 1" r3 LEFT JOIN "S 1"."T 1" r2 ON (((r2."C 1" = r3."C 1")))) ORDER BY r3."C 1" ASC NULLS LAST +(13 rows) SELECT t1."C 1", t2.c1, t3.c1 FROM "S 1"."T 1" t1 left join ft1 t2 full join ft2 t3 on (t2.c1 = t3.c1) on (t3.c1 = t1."C 1") OFFSET 100 LIMIT 10; C 1 | c1 | c1 @@ -781,7 +784,7 @@ EXPLAIN (VERBOSE, COSTS OFF) Index Cond: (a."C 1" = 47) -> Foreign Scan on public.ft2 b Output: b.c1, b.c2, b.c3, b.c4, b.c5, b.c6, b.c7, b.c8 - Remote SQL: SELECT "C 1", c2, c3, c4, c5, c6, c7, c8 FROM "S 1"."T 1" WHERE (("C 1" = $1::integer)) + Remote SQL: SELECT "C 1", c2, c3, c4, c5, c6, c7, c8 FROM "S 1"."T 1" WHERE (($1::integer = "C 1")) (8 rows) SELECT * FROM "S 1"."T 1" a, ft2 b WHERE a."C 1" = 47 AND b.c1 = a.c2; @@ -1419,7 +1422,7 @@ SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t Foreign Scan Output: t1.c1, t2.c1, t1.c3 Relations: (public.ft1 t1) INNER JOIN (public.ft2 t2) - Remote SQL: SELECT r1."C 1", r2."C 1", r1.c3 FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r2."C 1" = r1."C 1")))) ORDER BY r1.c3 ASC NULLS LAST, r1."C 1" ASC NULLS LAST LIMIT 10::bigint OFFSET 100::bigint + Remote SQL: SELECT r1."C 1", r2."C 1", r1.c3 FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r1."C 1" = r2."C 1")))) ORDER BY r1.c3 ASC NULLS LAST, r1."C 1" ASC NULLS LAST LIMIT 10::bigint OFFSET 100::bigint (4 rows) SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t1.c1 OFFSET 100 LIMIT 10; @@ -1445,7 +1448,7 @@ SELECT t1.c1, t2.c2, t3.c3 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) JOIN ft4 t Foreign Scan Output: t1.c1, t2.c2, t3.c3, t1.c3 Relations: ((public.ft1 t1) INNER JOIN (public.ft2 t2)) INNER JOIN (public.ft4 t3) - Remote SQL: SELECT r1."C 1", r2.c2, r4.c3, r1.c3 FROM (("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r2."C 1" = r1."C 1")))) INNER JOIN "S 1"."T 3" r4 ON (((r1."C 1" = r4.c1)))) ORDER BY r1.c3 ASC NULLS LAST, r1."C 1" ASC NULLS LAST LIMIT 10::bigint OFFSET 10::bigint + Remote SQL: SELECT r1."C 1", r2.c2, r4.c3, r1.c3 FROM (("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r1."C 1" = r2."C 1")))) INNER JOIN "S 1"."T 3" r4 ON (((r1."C 1" = r4.c1)))) ORDER BY r1.c3 ASC NULLS LAST, r1."C 1" ASC NULLS LAST LIMIT 10::bigint OFFSET 10::bigint (4 rows) SELECT t1.c1, t2.c2, t3.c3 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) JOIN ft4 t3 ON (t3.c1 = t1.c1) ORDER BY t1.c3, t1.c1 OFFSET 10 LIMIT 10; @@ -2060,7 +2063,7 @@ SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t Foreign Scan Output: t1.c1, t2.c1, t1.c3, t1.*, t2.* Relations: (public.ft1 t1) INNER JOIN (public.ft2 t2) - Remote SQL: SELECT r1."C 1", r2."C 1", r1.c3, CASE WHEN (r1.*)::text IS NOT NULL THEN ROW(r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8) END, CASE WHEN (r2.*)::text IS NOT NULL THEN ROW(r2."C 1", r2.c2, r2.c3, r2.c4, r2.c5, r2.c6, r2.c7, r2.c8) END FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r2."C 1" = r1."C 1")))) ORDER BY r1.c3 ASC NULLS LAST, r1."C 1" ASC NULLS LAST LIMIT 10::bigint OFFSET 100::bigint FOR UPDATE OF r1 + Remote SQL: SELECT r1."C 1", r2."C 1", r1.c3, CASE WHEN (r1.*)::text IS NOT NULL THEN ROW(r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8) END, CASE WHEN (r2.*)::text IS NOT NULL THEN ROW(r2."C 1", r2.c2, r2.c3, r2.c4, r2.c5, r2.c6, r2.c7, r2.c8) END FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r1."C 1" = r2."C 1")))) ORDER BY r1.c3 ASC NULLS LAST, r1."C 1" ASC NULLS LAST LIMIT 10::bigint OFFSET 100::bigint FOR UPDATE OF r1 (4 rows) SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t1.c1 OFFSET 100 LIMIT 10 FOR UPDATE OF t1; @@ -2085,7 +2088,7 @@ SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t Foreign Scan Output: t1.c1, t2.c1, t1.c3, t1.*, t2.* Relations: (public.ft1 t1) INNER JOIN (public.ft2 t2) - Remote SQL: SELECT r1."C 1", r2."C 1", r1.c3, CASE WHEN (r1.*)::text IS NOT NULL THEN ROW(r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8) END, CASE WHEN (r2.*)::text IS NOT NULL THEN ROW(r2."C 1", r2.c2, r2.c3, r2.c4, r2.c5, r2.c6, r2.c7, r2.c8) END FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r2."C 1" = r1."C 1")))) ORDER BY r1.c3 ASC NULLS LAST, r1."C 1" ASC NULLS LAST LIMIT 10::bigint OFFSET 100::bigint FOR UPDATE OF r1 FOR UPDATE OF r2 + Remote SQL: SELECT r1."C 1", r2."C 1", r1.c3, CASE WHEN (r1.*)::text IS NOT NULL THEN ROW(r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8) END, CASE WHEN (r2.*)::text IS NOT NULL THEN ROW(r2."C 1", r2.c2, r2.c3, r2.c4, r2.c5, r2.c6, r2.c7, r2.c8) END FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r1."C 1" = r2."C 1")))) ORDER BY r1.c3 ASC NULLS LAST, r1."C 1" ASC NULLS LAST LIMIT 10::bigint OFFSET 100::bigint FOR UPDATE OF r1 FOR UPDATE OF r2 (4 rows) SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t1.c1 OFFSET 100 LIMIT 10 FOR UPDATE; @@ -2111,7 +2114,7 @@ SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t Foreign Scan Output: t1.c1, t2.c1, t1.c3, t1.*, t2.* Relations: (public.ft1 t1) INNER JOIN (public.ft2 t2) - Remote SQL: SELECT r1."C 1", r2."C 1", r1.c3, CASE WHEN (r1.*)::text IS NOT NULL THEN ROW(r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8) END, CASE WHEN (r2.*)::text IS NOT NULL THEN ROW(r2."C 1", r2.c2, r2.c3, r2.c4, r2.c5, r2.c6, r2.c7, r2.c8) END FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r2."C 1" = r1."C 1")))) ORDER BY r1.c3 ASC NULLS LAST, r1."C 1" ASC NULLS LAST LIMIT 10::bigint OFFSET 100::bigint FOR SHARE OF r1 + Remote SQL: SELECT r1."C 1", r2."C 1", r1.c3, CASE WHEN (r1.*)::text IS NOT NULL THEN ROW(r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8) END, CASE WHEN (r2.*)::text IS NOT NULL THEN ROW(r2."C 1", r2.c2, r2.c3, r2.c4, r2.c5, r2.c6, r2.c7, r2.c8) END FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r1."C 1" = r2."C 1")))) ORDER BY r1.c3 ASC NULLS LAST, r1."C 1" ASC NULLS LAST LIMIT 10::bigint OFFSET 100::bigint FOR SHARE OF r1 (4 rows) SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t1.c1 OFFSET 100 LIMIT 10 FOR SHARE OF t1; @@ -2136,7 +2139,7 @@ SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t Foreign Scan Output: t1.c1, t2.c1, t1.c3, t1.*, t2.* Relations: (public.ft1 t1) INNER JOIN (public.ft2 t2) - Remote SQL: SELECT r1."C 1", r2."C 1", r1.c3, CASE WHEN (r1.*)::text IS NOT NULL THEN ROW(r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8) END, CASE WHEN (r2.*)::text IS NOT NULL THEN ROW(r2."C 1", r2.c2, r2.c3, r2.c4, r2.c5, r2.c6, r2.c7, r2.c8) END FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r2."C 1" = r1."C 1")))) ORDER BY r1.c3 ASC NULLS LAST, r1."C 1" ASC NULLS LAST LIMIT 10::bigint OFFSET 100::bigint FOR SHARE OF r1 FOR SHARE OF r2 + Remote SQL: SELECT r1."C 1", r2."C 1", r1.c3, CASE WHEN (r1.*)::text IS NOT NULL THEN ROW(r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8) END, CASE WHEN (r2.*)::text IS NOT NULL THEN ROW(r2."C 1", r2.c2, r2.c3, r2.c4, r2.c5, r2.c6, r2.c7, r2.c8) END FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r1."C 1" = r2."C 1")))) ORDER BY r1.c3 ASC NULLS LAST, r1."C 1" ASC NULLS LAST LIMIT 10::bigint OFFSET 100::bigint FOR SHARE OF r1 FOR SHARE OF r2 (4 rows) SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER BY t1.c3, t1.c1 OFFSET 100 LIMIT 10 FOR SHARE; @@ -2165,7 +2168,7 @@ WITH t (c1_1, c1_3, c2_1) AS MATERIALIZED (SELECT t1.c1, t1.c3, t2.c1 FROM ft1 t -> Foreign Scan Output: t1.c1, t1.c3, t2.c1 Relations: (public.ft1 t1) INNER JOIN (public.ft2 t2) - Remote SQL: SELECT r1."C 1", r1.c3, r2."C 1" FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r2."C 1" = r1."C 1")))) + Remote SQL: SELECT r1."C 1", r1.c3, r2."C 1" FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r1."C 1" = r2."C 1")))) -> Sort Output: t.c1_1, t.c2_1, t.c1_3 Sort Key: t.c1_3, t.c1_1 @@ -2196,7 +2199,7 @@ SELECT t1.ctid, t1, t2, t1.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) ORDER B Foreign Scan Output: t1.ctid, t1.*, t2.*, t1.c1, t1.c3 Relations: (public.ft1 t1) INNER JOIN (public.ft2 t2) - Remote SQL: SELECT r1.ctid, CASE WHEN (r1.*)::text IS NOT NULL THEN ROW(r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8) END, CASE WHEN (r2.*)::text IS NOT NULL THEN ROW(r2."C 1", r2.c2, r2.c3, r2.c4, r2.c5, r2.c6, r2.c7, r2.c8) END, r1."C 1", r1.c3 FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r2."C 1" = r1."C 1")))) ORDER BY r1.c3 ASC NULLS LAST, r1."C 1" ASC NULLS LAST LIMIT 10::bigint OFFSET 100::bigint + Remote SQL: SELECT r1.ctid, CASE WHEN (r1.*)::text IS NOT NULL THEN ROW(r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8) END, CASE WHEN (r2.*)::text IS NOT NULL THEN ROW(r2."C 1", r2.c2, r2.c3, r2.c4, r2.c5, r2.c6, r2.c7, r2.c8) END, r1."C 1", r1.c3 FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r1."C 1" = r2."C 1")))) ORDER BY r1.c3 ASC NULLS LAST, r1."C 1" ASC NULLS LAST LIMIT 10::bigint OFFSET 100::bigint (4 rows) -- SEMI JOIN @@ -2207,7 +2210,7 @@ SELECT t1.c1 FROM ft1 t1 WHERE EXISTS (SELECT 1 FROM ft2 t2 WHERE t1.c1 = t2.c1) Foreign Scan Output: t1.c1 Relations: (public.ft1 t1) SEMI JOIN (public.ft2 t2) - Remote SQL: SELECT r1."C 1" FROM "S 1"."T 1" r1 WHERE EXISTS (SELECT NULL FROM "S 1"."T 1" r2 WHERE ((r2."C 1" = r1."C 1"))) ORDER BY r1."C 1" ASC NULLS LAST LIMIT 10::bigint OFFSET 100::bigint + Remote SQL: SELECT r1."C 1" FROM "S 1"."T 1" r1 WHERE EXISTS (SELECT NULL FROM "S 1"."T 1" r2 WHERE ((r1."C 1" = r2."C 1"))) ORDER BY r1."C 1" ASC NULLS LAST LIMIT 10::bigint OFFSET 100::bigint (4 rows) SELECT t1.c1 FROM ft1 t1 WHERE EXISTS (SELECT 1 FROM ft2 t2 WHERE t1.c1 = t2.c1) ORDER BY t1.c1 OFFSET 100 LIMIT 10; @@ -2408,7 +2411,7 @@ SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) WHERE t1.c8 = t2. Output: t1.c1, t2.c1, t1.c3 Filter: (t1.c8 = t2.c8) Relations: (public.ft1 t1) INNER JOIN (public.ft2 t2) - Remote SQL: SELECT r1."C 1", r2."C 1", r1.c3, r1.c8, r2.c8 FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r2."C 1" = r1."C 1")))) + Remote SQL: SELECT r1."C 1", r2."C 1", r1.c3, r1.c8, r2.c8 FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r1."C 1" = r2."C 1")))) (10 rows) SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) WHERE t1.c8 = t2.c8 ORDER BY t1.c3, t1.c1 OFFSET 100 LIMIT 10; @@ -2446,11 +2449,11 @@ SELECT t1c1, avg(t1c1 + t2c1) FROM (SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 -> Foreign Scan Output: t1.c1, t2.c1 Relations: (public.ft1 t1) INNER JOIN (public.ft2 t2) - Remote SQL: SELECT r1."C 1", r2."C 1" FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r2."C 1" = r1."C 1")))) + Remote SQL: SELECT r1."C 1", r2."C 1" FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r1."C 1" = r2."C 1")))) -> Foreign Scan Output: t1_1.c1, t2_1.c1 Relations: (public.ft1 t1_1) INNER JOIN (public.ft2 t2_1) - Remote SQL: SELECT r1."C 1", r2."C 1" FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r2."C 1" = r1."C 1")))) + Remote SQL: SELECT r1."C 1", r2."C 1" FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r1."C 1" = r2."C 1")))) (20 rows) SELECT t1c1, avg(t1c1 + t2c1) FROM (SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1) UNION SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1)) AS t (t1c1, t2c1) GROUP BY t1c1 ORDER BY t1c1 OFFSET 100 LIMIT 10; @@ -2489,7 +2492,7 @@ SELECT t1."C 1" FROM "S 1"."T 1" t1, LATERAL (SELECT DISTINCT t2.c1, t3.c1 FROM -> Foreign Scan Output: t2.c1, t3.c1 Relations: (public.ft1 t2) INNER JOIN (public.ft2 t3) - Remote SQL: SELECT r1."C 1", r2."C 1" FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r2."C 1" = r1."C 1")) AND ((r1.c2 = $1::integer)))) + Remote SQL: SELECT r1."C 1", r2."C 1" FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r1."C 1" = r2."C 1")) AND ((r1.c2 = $1::integer)))) (17 rows) SELECT t1."C 1" FROM "S 1"."T 1" t1, LATERAL (SELECT DISTINCT t2.c1, t3.c1 FROM ft1 t2, ft2 t3 WHERE t2.c1 = t3.c1 AND t2.c2 = t1.c2) q ORDER BY t1."C 1" OFFSET 10 LIMIT 10; @@ -2520,7 +2523,7 @@ SELECT t1.c1, t2.c1 FROM ft1 t1 JOIN ft2 t2 ON (t1.c1 = t2.c1 AND CURRENT_USER = -> Foreign Scan Output: t1.c1, t1.c3, t2.c1 Relations: (public.ft1 t1) INNER JOIN (public.ft2 t2) - Remote SQL: SELECT r1."C 1", r2."C 1", r1.c3 FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r2."C 1" = r1."C 1")))) ORDER BY r1.c3 ASC NULLS LAST, r1."C 1" ASC NULLS LAST + Remote SQL: SELECT r1."C 1", r2."C 1", r1.c3 FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r1."C 1" = r2."C 1")))) ORDER BY r1.c3 ASC NULLS LAST, r1."C 1" ASC NULLS LAST (9 rows) -- non-Var items in targetlist of the nullable rel of a join preventing @@ -2612,56 +2615,57 @@ SET enable_hashjoin TO false; EXPLAIN (VERBOSE, COSTS OFF) SELECT * FROM ft1, ft2, ft4, ft5, local_tbl WHERE ft1.c1 = ft2.c1 AND ft1.c2 = ft4.c1 AND ft1.c2 = ft5.c1 AND ft1.c2 = local_tbl.c1 AND ft1.c1 < 100 AND ft2.c1 < 100 FOR UPDATE; - QUERY PLAN --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + QUERY PLAN +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- LockRows Output: ft1.c1, ft1.c2, ft1.c3, ft1.c4, ft1.c5, ft1.c6, ft1.c7, ft1.c8, ft2.c1, ft2.c2, ft2.c3, ft2.c4, ft2.c5, ft2.c6, ft2.c7, ft2.c8, ft4.c1, ft4.c2, ft4.c3, ft5.c1, ft5.c2, ft5.c3, local_tbl.c1, local_tbl.c2, local_tbl.c3, ft1.*, ft2.*, ft4.*, ft5.*, local_tbl.ctid -> Merge Join Output: ft1.c1, ft1.c2, ft1.c3, ft1.c4, ft1.c5, ft1.c6, ft1.c7, ft1.c8, ft2.c1, ft2.c2, ft2.c3, ft2.c4, ft2.c5, ft2.c6, ft2.c7, ft2.c8, ft4.c1, ft4.c2, ft4.c3, ft5.c1, ft5.c2, ft5.c3, local_tbl.c1, local_tbl.c2, local_tbl.c3, ft1.*, ft2.*, ft4.*, ft5.*, local_tbl.ctid - Inner Unique: true - Merge Cond: (ft1.c2 = local_tbl.c1) - -> Foreign Scan + Merge Cond: (local_tbl.c1 = ft1.c2) + -> Index Scan using local_tbl_pkey on public.local_tbl + Output: local_tbl.c1, local_tbl.c2, local_tbl.c3, local_tbl.ctid + -> Materialize Output: ft1.c1, ft1.c2, ft1.c3, ft1.c4, ft1.c5, ft1.c6, ft1.c7, ft1.c8, ft1.*, ft2.c1, ft2.c2, ft2.c3, ft2.c4, ft2.c5, ft2.c6, ft2.c7, ft2.c8, ft2.*, ft4.c1, ft4.c2, ft4.c3, ft4.*, ft5.c1, ft5.c2, ft5.c3, ft5.* - Relations: (((public.ft1) INNER JOIN (public.ft2)) INNER JOIN (public.ft4)) INNER JOIN (public.ft5) - Remote SQL: SELECT r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8, CASE WHEN (r1.*)::text IS NOT NULL THEN ROW(r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8) END, r2."C 1", r2.c2, r2.c3, r2.c4, r2.c5, r2.c6, r2.c7, r2.c8, CASE WHEN (r2.*)::text IS NOT NULL THEN ROW(r2."C 1", r2.c2, r2.c3, r2.c4, r2.c5, r2.c6, r2.c7, r2.c8) END, r3.c1, r3.c2, r3.c3, CASE WHEN (r3.*)::text IS NOT NULL THEN ROW(r3.c1, r3.c2, r3.c3) END, r4.c1, r4.c2, r4.c3, CASE WHEN (r4.*)::text IS NOT NULL THEN ROW(r4.c1, r4.c2, r4.c3) END FROM ((("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r2."C 1" = r1."C 1")) AND ((r2."C 1" < 100)) AND ((r1."C 1" < 100)))) INNER JOIN "S 1"."T 3" r3 ON (((r1.c2 = r3.c1)))) INNER JOIN "S 1"."T 4" r4 ON (((r1.c2 = r4.c1)))) ORDER BY r1.c2 ASC NULLS LAST FOR UPDATE OF r1 FOR UPDATE OF r2 FOR UPDATE OF r3 FOR UPDATE OF r4 - -> Merge Join + -> Foreign Scan Output: ft1.c1, ft1.c2, ft1.c3, ft1.c4, ft1.c5, ft1.c6, ft1.c7, ft1.c8, ft1.*, ft2.c1, ft2.c2, ft2.c3, ft2.c4, ft2.c5, ft2.c6, ft2.c7, ft2.c8, ft2.*, ft4.c1, ft4.c2, ft4.c3, ft4.*, ft5.c1, ft5.c2, ft5.c3, ft5.* - Merge Cond: (ft1.c2 = ft5.c1) + Relations: (((public.ft1) INNER JOIN (public.ft2)) INNER JOIN (public.ft4)) INNER JOIN (public.ft5) + Remote SQL: SELECT r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8, CASE WHEN (r1.*)::text IS NOT NULL THEN ROW(r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8) END, r2."C 1", r2.c2, r2.c3, r2.c4, r2.c5, r2.c6, r2.c7, r2.c8, CASE WHEN (r2.*)::text IS NOT NULL THEN ROW(r2."C 1", r2.c2, r2.c3, r2.c4, r2.c5, r2.c6, r2.c7, r2.c8) END, r3.c1, r3.c2, r3.c3, CASE WHEN (r3.*)::text IS NOT NULL THEN ROW(r3.c1, r3.c2, r3.c3) END, r4.c1, r4.c2, r4.c3, CASE WHEN (r4.*)::text IS NOT NULL THEN ROW(r4.c1, r4.c2, r4.c3) END FROM ((("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r1."C 1" = r2."C 1")) AND ((r2."C 1" < 100)) AND ((r1."C 1" < 100)))) INNER JOIN "S 1"."T 3" r3 ON (((r1.c2 = r3.c1)))) INNER JOIN "S 1"."T 4" r4 ON (((r1.c2 = r4.c1)))) ORDER BY r1.c2 ASC NULLS LAST FOR UPDATE OF r1 FOR UPDATE OF r2 FOR UPDATE OF r3 FOR UPDATE OF r4 -> Merge Join - Output: ft1.c1, ft1.c2, ft1.c3, ft1.c4, ft1.c5, ft1.c6, ft1.c7, ft1.c8, ft1.*, ft2.c1, ft2.c2, ft2.c3, ft2.c4, ft2.c5, ft2.c6, ft2.c7, ft2.c8, ft2.*, ft4.c1, ft4.c2, ft4.c3, ft4.* - Merge Cond: (ft1.c2 = ft4.c1) - -> Sort - Output: ft1.c1, ft1.c2, ft1.c3, ft1.c4, ft1.c5, ft1.c6, ft1.c7, ft1.c8, ft1.*, ft2.c1, ft2.c2, ft2.c3, ft2.c4, ft2.c5, ft2.c6, ft2.c7, ft2.c8, ft2.* - Sort Key: ft1.c2 - -> Merge Join + Output: ft1.c1, ft1.c2, ft1.c3, ft1.c4, ft1.c5, ft1.c6, ft1.c7, ft1.c8, ft1.*, ft2.c1, ft2.c2, ft2.c3, ft2.c4, ft2.c5, ft2.c6, ft2.c7, ft2.c8, ft2.*, ft4.c1, ft4.c2, ft4.c3, ft4.*, ft5.c1, ft5.c2, ft5.c3, ft5.* + Merge Cond: (ft1.c2 = ft5.c1) + -> Merge Join + Output: ft1.c1, ft1.c2, ft1.c3, ft1.c4, ft1.c5, ft1.c6, ft1.c7, ft1.c8, ft1.*, ft2.c1, ft2.c2, ft2.c3, ft2.c4, ft2.c5, ft2.c6, ft2.c7, ft2.c8, ft2.*, ft4.c1, ft4.c2, ft4.c3, ft4.* + Merge Cond: (ft1.c2 = ft4.c1) + -> Sort Output: ft1.c1, ft1.c2, ft1.c3, ft1.c4, ft1.c5, ft1.c6, ft1.c7, ft1.c8, ft1.*, ft2.c1, ft2.c2, ft2.c3, ft2.c4, ft2.c5, ft2.c6, ft2.c7, ft2.c8, ft2.* - Merge Cond: (ft1.c1 = ft2.c1) - -> Sort - Output: ft1.c1, ft1.c2, ft1.c3, ft1.c4, ft1.c5, ft1.c6, ft1.c7, ft1.c8, ft1.* - Sort Key: ft1.c1 - -> Foreign Scan on public.ft1 + Sort Key: ft1.c2 + -> Merge Join + Output: ft1.c1, ft1.c2, ft1.c3, ft1.c4, ft1.c5, ft1.c6, ft1.c7, ft1.c8, ft1.*, ft2.c1, ft2.c2, ft2.c3, ft2.c4, ft2.c5, ft2.c6, ft2.c7, ft2.c8, ft2.* + Merge Cond: (ft1.c1 = ft2.c1) + -> Sort Output: ft1.c1, ft1.c2, ft1.c3, ft1.c4, ft1.c5, ft1.c6, ft1.c7, ft1.c8, ft1.* - Remote SQL: SELECT "C 1", c2, c3, c4, c5, c6, c7, c8 FROM "S 1"."T 1" WHERE (("C 1" < 100)) FOR UPDATE - -> Materialize - Output: ft2.c1, ft2.c2, ft2.c3, ft2.c4, ft2.c5, ft2.c6, ft2.c7, ft2.c8, ft2.* - -> Foreign Scan on public.ft2 + Sort Key: ft1.c1 + -> Foreign Scan on public.ft1 + Output: ft1.c1, ft1.c2, ft1.c3, ft1.c4, ft1.c5, ft1.c6, ft1.c7, ft1.c8, ft1.* + Remote SQL: SELECT "C 1", c2, c3, c4, c5, c6, c7, c8 FROM "S 1"."T 1" WHERE (("C 1" < 100)) FOR UPDATE + -> Materialize Output: ft2.c1, ft2.c2, ft2.c3, ft2.c4, ft2.c5, ft2.c6, ft2.c7, ft2.c8, ft2.* - Remote SQL: SELECT "C 1", c2, c3, c4, c5, c6, c7, c8 FROM "S 1"."T 1" WHERE (("C 1" < 100)) ORDER BY "C 1" ASC NULLS LAST FOR UPDATE - -> Sort - Output: ft4.c1, ft4.c2, ft4.c3, ft4.* - Sort Key: ft4.c1 - -> Foreign Scan on public.ft4 + -> Foreign Scan on public.ft2 + Output: ft2.c1, ft2.c2, ft2.c3, ft2.c4, ft2.c5, ft2.c6, ft2.c7, ft2.c8, ft2.* + Remote SQL: SELECT "C 1", c2, c3, c4, c5, c6, c7, c8 FROM "S 1"."T 1" WHERE (("C 1" < 100)) ORDER BY "C 1" ASC NULLS LAST FOR UPDATE + -> Sort Output: ft4.c1, ft4.c2, ft4.c3, ft4.* - Remote SQL: SELECT c1, c2, c3 FROM "S 1"."T 3" FOR UPDATE - -> Sort - Output: ft5.c1, ft5.c2, ft5.c3, ft5.* - Sort Key: ft5.c1 - -> Foreign Scan on public.ft5 + Sort Key: ft4.c1 + -> Foreign Scan on public.ft4 + Output: ft4.c1, ft4.c2, ft4.c3, ft4.* + Remote SQL: SELECT c1, c2, c3 FROM "S 1"."T 3" FOR UPDATE + -> Sort Output: ft5.c1, ft5.c2, ft5.c3, ft5.* - Remote SQL: SELECT c1, c2, c3 FROM "S 1"."T 4" FOR UPDATE - -> Index Scan using local_tbl_pkey on public.local_tbl - Output: local_tbl.c1, local_tbl.c2, local_tbl.c3, local_tbl.ctid -(47 rows) + Sort Key: ft5.c1 + -> Foreign Scan on public.ft5 + Output: ft5.c1, ft5.c2, ft5.c3, ft5.* + Remote SQL: SELECT c1, c2, c3 FROM "S 1"."T 4" FOR UPDATE +(48 rows) SELECT * FROM ft1, ft2, ft4, ft5, local_tbl WHERE ft1.c1 = ft2.c1 AND ft1.c2 = ft4.c1 AND ft1.c2 = ft5.c1 AND ft1.c2 = local_tbl.c1 AND ft1.c1 < 100 AND ft2.c1 < 100 FOR UPDATE; @@ -2763,7 +2767,7 @@ SELECT * FROM local_tbl LEFT JOIN (SELECT ft1.*, COALESCE(ft1.c3 || ft2.c3, 'foo -> Foreign Scan Output: ft1.c1, ft1.c2, ft1.c3, ft1.c4, ft1.c5, ft1.c6, ft1.c7, ft1.c8, ft1.*, ft2.*, COALESCE((ft1.c3 || ft2.c3), 'foobar'::text) Relations: (public.ft1) INNER JOIN (public.ft2) - Remote SQL: SELECT r4."C 1", r4.c2, r4.c3, r4.c4, r4.c5, r4.c6, r4.c7, r4.c8, CASE WHEN (r4.*)::text IS NOT NULL THEN ROW(r4."C 1", r4.c2, r4.c3, r4.c4, r4.c5, r4.c6, r4.c7, r4.c8) END, CASE WHEN (r5.*)::text IS NOT NULL THEN ROW(r5."C 1", r5.c2, r5.c3, r5.c4, r5.c5, r5.c6, r5.c7, r5.c8) END, r5.c3 FROM ("S 1"."T 1" r4 INNER JOIN "S 1"."T 1" r5 ON (((r5."C 1" = r4."C 1")) AND ((r4."C 1" < 100)))) ORDER BY r4."C 1" ASC NULLS LAST + Remote SQL: SELECT r4."C 1", r4.c2, r4.c3, r4.c4, r4.c5, r4.c6, r4.c7, r4.c8, CASE WHEN (r4.*)::text IS NOT NULL THEN ROW(r4."C 1", r4.c2, r4.c3, r4.c4, r4.c5, r4.c6, r4.c7, r4.c8) END, CASE WHEN (r5.*)::text IS NOT NULL THEN ROW(r5."C 1", r5.c2, r5.c3, r5.c4, r5.c5, r5.c6, r5.c7, r5.c8) END, r5.c3 FROM ("S 1"."T 1" r4 INNER JOIN "S 1"."T 1" r5 ON (((r4."C 1" = r5."C 1")) AND ((r4."C 1" < 100)))) ORDER BY r4."C 1" ASC NULLS LAST -> Result Output: ft1.c1, ft1.c2, ft1.c3, ft1.c4, ft1.c5, ft1.c6, ft1.c7, ft1.c8, ft1.*, ft2.*, ft2.c3 -> Sort @@ -2801,7 +2805,7 @@ SELECT * FROM local_tbl LEFT JOIN (SELECT ft1.* FROM ft1 INNER JOIN ft2 ON (ft1. Output: ft1.c1, ft1.c2, ft1.c3, ft1.c4, ft1.c5, ft1.c6, ft1.c7, ft1.c8, ft1.*, ft2.* Filter: ((ft1.c1 - postgres_fdw_abs(ft2.c2)) = 0) Relations: (public.ft1) INNER JOIN (public.ft2) - Remote SQL: SELECT r4."C 1", r4.c2, r4.c3, r4.c4, r4.c5, r4.c6, r4.c7, r4.c8, CASE WHEN (r4.*)::text IS NOT NULL THEN ROW(r4."C 1", r4.c2, r4.c3, r4.c4, r4.c5, r4.c6, r4.c7, r4.c8) END, CASE WHEN (r5.*)::text IS NOT NULL THEN ROW(r5."C 1", r5.c2, r5.c3, r5.c4, r5.c5, r5.c6, r5.c7, r5.c8) END, r5.c2 FROM ("S 1"."T 1" r4 INNER JOIN "S 1"."T 1" r5 ON (((r5."C 1" = r4."C 1")) AND ((r4."C 1" < 100)))) ORDER BY r4.c3 ASC NULLS LAST + Remote SQL: SELECT r4."C 1", r4.c2, r4.c3, r4.c4, r4.c5, r4.c6, r4.c7, r4.c8, CASE WHEN (r4.*)::text IS NOT NULL THEN ROW(r4."C 1", r4.c2, r4.c3, r4.c4, r4.c5, r4.c6, r4.c7, r4.c8) END, CASE WHEN (r5.*)::text IS NOT NULL THEN ROW(r5."C 1", r5.c2, r5.c3, r5.c4, r5.c5, r5.c6, r5.c7, r5.c8) END, r5.c2 FROM ("S 1"."T 1" r4 INNER JOIN "S 1"."T 1" r5 ON (((r4."C 1" = r5."C 1")) AND ((r4."C 1" < 100)))) ORDER BY r4.c3 ASC NULLS LAST -> Sort Output: ft1.c1, ft1.c2, ft1.c3, ft1.c4, ft1.c5, ft1.c6, ft1.c7, ft1.c8, ft1.*, ft2.*, ft2.c2 Sort Key: ft1.c3 @@ -3085,7 +3089,7 @@ select sum(t1.c1), count(t2.c1) from ft1 t1 inner join ft2 t2 on (t1.c1 = t2.c1) Output: t1.c1 Filter: (((((t1.c1 * t2.c1) / (t1.c1 * t2.c1)))::double precision * random()) <= '1'::double precision) Relations: (public.ft1 t1) INNER JOIN (public.ft2 t2) - Remote SQL: SELECT r1."C 1", r2."C 1" FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r2."C 1" = r1."C 1")))) + Remote SQL: SELECT r1."C 1", r2."C 1" FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r1."C 1" = r2."C 1")))) (7 rows) -- GROUP BY clause having expressions @@ -4295,7 +4299,7 @@ EXPLAIN (VERBOSE, COSTS OFF) EXECUTE st2(10, 20); ---------------------------------------------------------------------------------------------------------------------------------- Nested Loop Semi Join Output: t1.c1, t1.c2, t1.c3, t1.c4, t1.c5, t1.c6, t1.c7, t1.c8 - Join Filter: (t2.c3 = t1.c3) + Join Filter: (t1.c3 = t2.c3) -> Foreign Scan on public.ft1 t1 Output: t1.c1, t1.c2, t1.c3, t1.c4, t1.c5, t1.c6, t1.c7, t1.c8 Remote SQL: SELECT "C 1", c2, c3, c4, c5, c6, c7, c8 FROM "S 1"."T 1" WHERE (("C 1" < 20)) ORDER BY "C 1" ASC NULLS LAST @@ -4328,7 +4332,7 @@ EXPLAIN (VERBOSE, COSTS OFF) EXECUTE st3(10, 20); Foreign Scan Output: t1.c1, t1.c2, t1.c3, t1.c4, t1.c5, t1.c6, t1.c7, t1.c8 Relations: (public.ft1 t1) SEMI JOIN (public.ft2 t2) - Remote SQL: SELECT r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8 FROM "S 1"."T 1" r1 WHERE ((r1."C 1" < 20)) AND EXISTS (SELECT NULL FROM "S 1"."T 1" r3 WHERE ((r3."C 1" > 10)) AND ((date(r3.c5) = '1970-01-17'::date)) AND ((r3.c3 = r1.c3))) ORDER BY r1."C 1" ASC NULLS LAST + Remote SQL: SELECT r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8 FROM "S 1"."T 1" r1 WHERE ((r1."C 1" < 20)) AND EXISTS (SELECT NULL FROM "S 1"."T 1" r3 WHERE ((r3."C 1" > 10)) AND ((date(r3.c5) = '1970-01-17'::date)) AND ((r1.c3 = r3.c3))) ORDER BY r1."C 1" ASC NULLS LAST (4 rows) EXECUTE st3(10, 20); @@ -5113,7 +5117,7 @@ EXPLAIN (verbose, costs off) Foreign Scan Output: ft2.c1, ft2.c2, ft2.c3, ft2.c4, ft2.c5, ft2.c6, ft2.c7, ft2.c8, ft4.c1, ft4.c2, ft4.c3 Relations: ((((public.ft2) INNER JOIN (public.ft4)) SEMI JOIN (public.ft2 ft2_1)) INNER JOIN (public.ft2 ft2_2)) SEMI JOIN (public.ft4 ft4_1) - Remote SQL: SELECT r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8, r6.c1, r6.c2, r6.c3 FROM (("S 1"."T 1" r1 INNER JOIN "S 1"."T 3" r6 ON (((r1.c2 = r6.c1)) AND ((r1."C 1" > 900)))) INNER JOIN "S 1"."T 1" r8 ON (((r1.c2 = r8.c2)))) WHERE EXISTS (SELECT NULL FROM "S 1"."T 3" r9 WHERE ((r1.c2 = r9.c2))) AND EXISTS (SELECT NULL FROM "S 1"."T 1" r7 WHERE ((r7.c2 = r6.c2))) ORDER BY r1."C 1" ASC NULLS LAST LIMIT 10::bigint + Remote SQL: SELECT r1."C 1", r1.c2, r1.c3, r1.c4, r1.c5, r1.c6, r1.c7, r1.c8, r6.c1, r6.c2, r6.c3 FROM (("S 1"."T 1" r1 INNER JOIN "S 1"."T 3" r6 ON (((r1.c2 = r6.c1)) AND ((r1."C 1" > 900)))) INNER JOIN "S 1"."T 1" r8 ON (((r1.c2 = r8.c2)))) WHERE EXISTS (SELECT NULL FROM "S 1"."T 3" r9 WHERE ((r1.c2 = r9.c2))) AND EXISTS (SELECT NULL FROM "S 1"."T 1" r7 WHERE ((r6.c2 = r7.c2))) ORDER BY r1."C 1" ASC NULLS LAST LIMIT 10::bigint (4 rows) SELECT ft2.*, ft4.* FROM ft2 INNER JOIN @@ -5184,7 +5188,7 @@ SELECT ft1.c1 FROM ft1 JOIN ft2 on ft1.c1 = ft2.c1 WHERE -> Foreign Scan Output: ft1.c1, ft2.c1 Relations: (public.ft1) INNER JOIN (public.ft2) - Remote SQL: SELECT r1."C 1", r2."C 1" FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r2."C 1" = r1."C 1")))) ORDER BY r1."C 1" ASC NULLS LAST + Remote SQL: SELECT r1."C 1", r2."C 1" FROM ("S 1"."T 1" r1 INNER JOIN "S 1"."T 1" r2 ON (((r1."C 1" = r2."C 1")))) ORDER BY r1."C 1" ASC NULLS LAST -> Foreign Scan Output: ft2_1.c1, ft4.c1 Relations: (public.ft2 ft2_1) INNER JOIN (public.ft4) diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index a40d03d35f3..eb92d0e094b 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -25,6 +25,7 @@ #include "commands/prepare.h" #include "foreign/fdwapi.h" #include "jit/jit.h" +#include "lib/bloomfilter.h" #include "libpq/pqformat.h" #include "libpq/protocol.h" #include "nodes/extensible.h" @@ -92,6 +93,8 @@ static void show_scan_qual(List *qual, const char *qlabel, static void show_upper_qual(List *qual, const char *qlabel, PlanState *planstate, List *ancestors, ExplainState *es); +static void show_bloom_filter_info(PlanState *planstate, List *ancestors, + ExplainState *es); static void show_sort_keys(SortState *sortstate, List *ancestors, ExplainState *es); static void show_incremental_sort_keys(IncrementalSortState *incrsortstate, @@ -1978,6 +1981,7 @@ ExplainNode(PlanState *planstate, List *ancestors, if (plan->qual) show_instrumentation_count("Rows Removed by Filter", 1, planstate, es); + show_bloom_filter_info(planstate, ancestors, es); show_indexsearches_info(planstate, es); break; case T_IndexOnlyScan: @@ -1995,6 +1999,7 @@ ExplainNode(PlanState *planstate, List *ancestors, if (es->analyze) ExplainPropertyFloat("Heap Fetches", NULL, planstate->instrument->ntuples2, 0, es); + show_bloom_filter_info(planstate, ancestors, es); show_indexsearches_info(planstate, es); break; case T_BitmapIndexScan: @@ -2012,6 +2017,7 @@ ExplainNode(PlanState *planstate, List *ancestors, if (plan->qual) show_instrumentation_count("Rows Removed by Filter", 1, planstate, es); + show_bloom_filter_info(planstate, ancestors, es); show_tidbitmap_info((BitmapHeapScanState *) planstate, es); show_scan_io_usage((ScanState *) planstate, es); break; @@ -2030,6 +2036,7 @@ ExplainNode(PlanState *planstate, List *ancestors, if (plan->qual) show_instrumentation_count("Rows Removed by Filter", 1, planstate, es); + show_bloom_filter_info(planstate, ancestors, es); if (IsA(plan, CteScan)) show_ctescan_info(castNode(CteScanState, planstate), es); show_scan_io_usage((ScanState *) planstate, es); @@ -2132,6 +2139,7 @@ ExplainNode(PlanState *planstate, List *ancestors, if (plan->qual) show_instrumentation_count("Rows Removed by Filter", 1, planstate, es); + show_bloom_filter_info(planstate, ancestors, es); } break; case T_TidRangeScan: @@ -2149,6 +2157,7 @@ ExplainNode(PlanState *planstate, List *ancestors, if (plan->qual) show_instrumentation_count("Rows Removed by Filter", 1, planstate, es); + show_bloom_filter_info(planstate, ancestors, es); show_scan_io_usage((ScanState *) planstate, es); } break; @@ -2578,6 +2587,135 @@ show_upper_qual(List *qual, const char *qlabel, show_qual(qual, qlabel, planstate, ancestors, useprefix, es); } +/* + * show_bloom_filter_info + * Show info about every Bloom filter pushed down to a scan node. + * + * In TEXT format each filter is rendered on a single line, e.g. + * + * Bloom Filter N: keys=(a, b) checked=99999 rejected=99990 + * + * The checked/rejected fields are omitted outside of ANALYZE). In structured + * formats we emit a group per filter with the same fields broken out as + * properties. + * + * Called from the per-recipient cases in ExplainNode; the recipient is expected + * to be a scan node, but nothing here depends on that. We may choose to push + * filters to other nodes in the plan. + * + * This only prints information about the keys, and the checked/rejected counts. + * Detailed information about the filter itself (number of bits, number of hash + * functions, ...) is available on the producer side. + * + * XXX It might be a good idea to show the expected filter selectivity too, + * not just the one actually observed during execution. That'd make it easier + * to reason about the decisions and review the plan changes. + * + * XXX If we choose other filter types, we'd need some general way to show the + * information. I don't know if we should have a "generic" information provided + * by all the filters, or if we would need a way to print custom information. + * Chances are we'd have a limited number of supported filter types, in which + * case we can have a show_ function for each type. Only if users could inject + * arbitrary filters, that'd be an issue. But that seems unlikely. + */ +static void +show_bloom_filter_info(PlanState *planstate, List *ancestors, + ExplainState *es) +{ + Plan *plan = planstate->plan; + ListCell *lc1, + *lc2; + List *deparse_cxt = NIL; + bool useprefix = false; + + if (plan->bloom_filters == NIL) + return; + + deparse_cxt = set_deparse_context_plan(es->deparse_cxt, + plan, ancestors); + useprefix = (IsA(plan, SubqueryScan) || es->verbose); + + if (es->format != EXPLAIN_FORMAT_TEXT) + ExplainOpenGroup("Bloom Filters", "Bloom Filters", false, es); + + /* print info about all the bloom filters */ + forboth(lc1, plan->bloom_filters, + lc2, planstate->bloom_filters) + { + BloomFilter *bf = lfirst_node(BloomFilter, lc1); + BloomFilterState *bfs = (BloomFilterState *) lfirst(lc2); + ListCell *lc; + StringInfoData keys; + bool first = true; + + initStringInfo(&keys); + + /* deparse the filter expressions */ + appendStringInfoChar(&keys, '('); + foreach(lc, bf->filter_exprs) + { + char *key; + Node *expr = (Node *) lfirst(lc); + + if (!first) + appendStringInfoString(&keys, ", "); + key = deparse_expression(expr, deparse_cxt, + useprefix, false); + appendStringInfoString(&keys, key); + first = false; + } + appendStringInfoChar(&keys, ')'); + + if (es->format == EXPLAIN_FORMAT_TEXT) + { + StringInfoData buf; + + initStringInfo(&buf); + + /* show the filter ID only when there are more filters */ + appendStringInfo(&buf, "Bloom Filter %d: keys=%s", + bf->producer_id, keys.data); + + /* include the counts only during ANALYZE */ + if (es->analyze && bfs != NULL) + { + /* rejected fraction */ + double frac = 100.0 * bfs->rejected / Max(1, bfs->checked); + + appendStringInfo(&buf, + " checked=" UINT64_FORMAT + " rejected=" UINT64_FORMAT + " (%.1f%%)", + bfs->checked, bfs->rejected, frac); + } + + ExplainIndentText(es); + appendStringInfoString(es->str, buf.data); + appendStringInfoChar(es->str, '\n'); + pfree(buf.data); + } + else /* non-text format */ + { + ExplainOpenGroup("Bloom Filter", NULL, true, es); + ExplainPropertyText("Keys", keys.data, es); + ExplainPropertyInteger("ID", NULL, bf->producer_id, es); + if (es->analyze && bfs != NULL) + { + ExplainPropertyFloat("Checked", NULL, + (double) bfs->checked, 0, es); + ExplainPropertyFloat("Rejected", NULL, + (double) bfs->rejected, 0, es); + } + ExplainCloseGroup("Bloom Filter", NULL, true, es); + } + + pfree(keys.data); + } + + if (es->format != EXPLAIN_FORMAT_TEXT) + ExplainCloseGroup("Bloom Filters", "Bloom Filters", false, es); +} + /* * Show the sort keys for a Sort node. */ @@ -3474,6 +3612,84 @@ show_hash_info(HashState *hashstate, ExplainState *es) spacePeakKb); } } + + /* + * Show infromation about the Bloom filter produced by this Hash node (if + * any). For plain EXPLAIN, the filter is not initialized / built, but we + * still show available plan-time metadata (at least the ID). + * + * Once the filter is built, we show the various filter parameters (number + * of bits, hash functions, ...). Note that even with EXPLAIN ANALYZE the + * filter may not be built, due to the hashjoin delaying the Hash build + * until on the first outer tuple. + * + * XXX We don't show the keys, because those are always the join keys. + * + * XXX I think it makes sense to show the checked/rejected counters both + * here and for the consumer. If/when we allow multiple consumers for the + * filter, then this would show the "summary" while the scan node would + * show just counters for that one consumer. + */ + if (hashstate->bloom_filter_id > 0) + { + int producer_id = hashstate->bloom_filter_id; + uint64 nbits = 0; + int nhashfns = 0; + uint64 bytes = 0; + + if (hashstate->bloom_filter != NULL) + { + nbits = bloom_total_bits(hashstate->bloom_filter); + nhashfns = bloom_hash_funcs(hashstate->bloom_filter); + bytes = nbits / 8; + } + + if (es->format == EXPLAIN_FORMAT_TEXT) + { + ExplainIndentText(es); + if (hashstate->bloom_filter != NULL) + appendStringInfo(es->str, + "Bloom Filter %d: bits=" UINT64_FORMAT + " hashes=%d memory=" UINT64_FORMAT "kB" + " checked=" UINT64_FORMAT " rejected=" UINT64_FORMAT "\n", + producer_id, + nbits, + nhashfns, + BYTES_TO_KILOBYTES(bytes), + hashstate->bloomFilterChecked, + hashstate->bloomFilterRejected); + else if (es->analyze) + appendStringInfo(es->str, + "Bloom Filter %d: (not initialized)\n", + producer_id); + else + appendStringInfo(es->str, + "Bloom Filter %d\n", + producer_id); + } + else + { + /* there can be just one bloom filter per fproducer (for now) */ + ExplainOpenGroup("Bloom Filter", "Bloom Filter", true, es); + + ExplainPropertyInteger("Producer", NULL, producer_id, es); + ExplainPropertyBool("Initialized", + (hashstate->bloom_filter != NULL), es); + + if (hashstate->bloom_filter != NULL) + { + ExplainPropertyUInteger("Bits", NULL, nbits, es); + ExplainPropertyInteger("Hash Functions", NULL, nhashfns, es); + ExplainPropertyUInteger("Memory Usage", "kB", + BYTES_TO_KILOBYTES(bytes), es); + ExplainPropertyFloat("Checked", NULL, + (double) hashstate->bloomFilterChecked, 0, es); + ExplainPropertyFloat("Rejected", NULL, + (double) hashstate->bloomFilterRejected, 0, es); + } + ExplainCloseGroup("Bloom Filter", "Bloom Filter", true, es); + } + } } /* diff --git a/src/backend/executor/execUtils.c b/src/backend/executor/execUtils.c index 1eb6b9f1f40..3c32a61dddd 100644 --- a/src/backend/executor/execUtils.c +++ b/src/backend/executor/execUtils.c @@ -159,6 +159,8 @@ CreateExecutorState(void) estate->es_auxmodifytables = NIL; + estate->es_bloom_producers = NIL; + estate->es_per_tuple_exprcontext = NULL; estate->es_sourceText = NULL; diff --git a/src/backend/executor/nodeBitmapHeapscan.c b/src/backend/executor/nodeBitmapHeapscan.c index 83d6478bc2b..e71a47b6205 100644 --- a/src/backend/executor/nodeBitmapHeapscan.c +++ b/src/backend/executor/nodeBitmapHeapscan.c @@ -456,6 +456,9 @@ ExecInitBitmapHeapScan(BitmapHeapScan *node, EState *estate, int eflags) scanstate->bitmapqualorig = ExecInitQual(node->bitmapqualorig, (PlanState *) scanstate); + ExecInitBloomFilters((PlanState *) scanstate, + scanstate->ss.ss_ScanTupleSlot); + scanstate->ss.ss_currentRelation = currentRelation; /* diff --git a/src/backend/executor/nodeHash.c b/src/backend/executor/nodeHash.c index 8825bb6fa23..3d84fec8527 100644 --- a/src/backend/executor/nodeHash.c +++ b/src/backend/executor/nodeHash.c @@ -35,6 +35,7 @@ #include "executor/instrument.h" #include "executor/nodeHash.h" #include "executor/nodeHashjoin.h" +#include "lib/bloomfilter.h" #include "miscadmin.h" #include "port/pg_bitutils.h" #include "utils/lsyscache.h" @@ -184,6 +185,18 @@ MultiExecPrivateHash(HashState *node) uint32 hashvalue = DatumGetUInt32(hashdatum); int bucketNumber; + /* + * Add the tuple to the pushed-down bloom filter (if any). Do it + * here (rather than in ExecHashTableInsert) so that each tuple is + * added exactly once, even if it later gets shuffled between + * batches by ExecHashIncreaseNumBatches. The filter would still + * produce the same matches, but it costs CPU. + */ + if (node->bloom_filter != NULL) + bloom_add_element(node->bloom_filter, + (unsigned char *) &hashvalue, + sizeof(hashvalue)); + bucketNumber = ExecHashGetSkewBucket(hashtable, hashvalue); if (bucketNumber != INVALID_SKEW_BUCKET_NO) { @@ -665,6 +678,53 @@ ExecHashTableCreate(HashState *state) MemoryContextSwitchTo(oldcxt); } + /* + * If we managed to push down a bloom filter to the outer side of the hash + * join, allocate it with the hash table. + * + * Whether we build the filter is decided by try_push_bloom_filter at plan + * time. If there's no recipient node, or when the GUC is set to off, + * state->want_bloom_filter is false. + * + * XXX We don't do this for parallel hash joins, to keep the PoC simple. + * The filter would need to live in shared memory, and the workers would + * need to coordinate to build it. But it's doable. + * + * The filter lives in the HashState, in the hashCtx memory context. That + * means it gets destroyed along with the hashtable, and it follows the + * same lifecycle (during rescans, etc.). + * + * The size of the filter is bounded by both the estimated inner row count + * and a fixed fraction of work_mem. bloom_create() will round down to + * the next power-of-two bitset and enforces a 1MB minimum. + * + * XXX This may need more thought. If we limit bloom_work_mem too much, + * the false positive rate will get too bad, and we won't filter enough + * tuples for the filter to pay for itself. The adaptive behavior will + * eventually skip the filter, but we could just not build it at all? Or + * do we want to take the chance, sometimes? + */ + if (state->want_bloom_filter) + { + MemoryContext oldctx; + int bloom_work_mem; + + /* only serial hashjoins for now, init only once */ + Assert(hashtable->parallel_state == NULL); + Assert(state->bloom_filter == NULL); + + state->bloomFilterChecked = 0; + state->bloomFilterRejected = 0; + + /* Cap bloom filter at ~1/8 of work_mem, but not less than 1MB. */ + bloom_work_mem = Max(1024, work_mem / 8); + + oldctx = MemoryContextSwitchTo(hashtable->hashCxt); + state->bloom_filter = bloom_create((int64) Max(rows, 1.0), + bloom_work_mem, 0); + MemoryContextSwitchTo(oldctx); + } + return hashtable; } diff --git a/src/backend/executor/nodeHashjoin.c b/src/backend/executor/nodeHashjoin.c index 202dd866251..c3cb57e0322 100644 --- a/src/backend/executor/nodeHashjoin.c +++ b/src/backend/executor/nodeHashjoin.c @@ -169,7 +169,9 @@ #include "executor/instrument.h" #include "executor/nodeHash.h" #include "executor/nodeHashjoin.h" +#include "lib/bloomfilter.h" #include "miscadmin.h" +#include "port/pg_bitutils.h" #include "utils/lsyscache.h" #include "utils/sharedtuplestore.h" #include "utils/tuplestore.h" @@ -834,6 +836,7 @@ HashJoinState * ExecInitHashJoin(HashJoin *node, EState *estate, int eflags) { HashJoinState *hjstate; + HashState *hashState; Plan *outerNode; Hash *hashNode; TupleDesc outerDesc, @@ -875,11 +878,37 @@ ExecInitHashJoin(HashJoin *node, EState *estate, int eflags) outerNode = outerPlan(node); hashNode = (Hash *) innerPlan(node); + /* + * Register ourselves as a bloom-filter producer in the EState before + * recursing into the outer subtree, so the scan node (we pushed the + * filter to) can find us. We do this only if we actually managed to push + * down the filter to a scan node. + */ + if (node->bloom_consumer_count > 0) + ExecRegisterBloomFilterProducer(hjstate); + outerPlanState(hjstate) = ExecInitNode(outerNode, estate, eflags); outerDesc = ExecGetResultType(outerPlanState(hjstate)); innerPlanState(hjstate) = ExecInitNode((Plan *) hashNode, estate, eflags); innerDesc = ExecGetResultType(innerPlanState(hjstate)); + /* + * Tell the Hash child to actually build the bloom filter, and the ID + * assigned to the filter. + * + * XXX Seems a bit ugly to manipulate the inner plan state like this. + * Surely there's a better way. OTOH the two nodes are pretty tightly + * coupled already, so maybe it's fine. + * + * XXX Also, this assumes the hash table is not built by ExecInitNode(), + * which is true for now. But maybe we will relax that in the future (e.g. + * so that the scan can push the filter to storage / to remote FDW node / + * ...)? + */ + hashState = castNode(HashState, innerPlanState(hjstate)); + hashState->want_bloom_filter = (node->bloom_consumer_count > 0); + hashState->bloom_filter_id = node->bloom_filter_id; + /* * Initialize result slot, type and projection. */ @@ -1080,11 +1109,15 @@ ExecEndHashJoin(HashJoinState *node) /* * Free hash table + * + * Clear the bloom_filter pointer. It lives in hashCxt, so it gets freed + * by the ExecHashTableDestroy call. */ if (node->hj_HashTable) { ExecHashTableDestroy(node->hj_HashTable); node->hj_HashTable = NULL; + hashNode->bloom_filter = NULL; } /* @@ -1737,6 +1770,12 @@ ExecReScanHashJoin(HashJoinState *node) node->hj_HashTable = NULL; node->hj_JoinState = HJ_BUILD_HASHTABLE; + /* + * Clear the bloom_filter pointer. It lives in hashCxt, so it gets + * freed by the ExecHashTableDestroy call. + */ + hashNode->bloom_filter = NULL; + /* * if chgParam of subnode is not null then plan will be re-scanned * by first ExecProcNode. @@ -1975,3 +2014,420 @@ ExecHashJoinInitializeWorker(HashJoinState *state, ExecSetExecProcNode(&state->js.ps, ExecParallelHashJoin); } + +/* + * BLOOM FILTER PUSHDOWN + * + * The pushdown decision is done in try_push_bloom_filter, when constructing + * the plan from the selected paths. The input paths track filters "expected" + * by scan nodes included in that path (if any). The planner then ensures all + * expected filters are either satisfied (by matching joins) or propagated up. + * In a valid plan all expected filters are satisfied. + * + * Then at execution time: + * + * - ExecInitHashJoin registers itself in EState.es_bloom_producers + * before recursing into child plans, so by the time a recipient's + * ExecInit runs, the producer is already discoverable by filter ID. + * This registration only happens when there's at least one consumer. + * It also sets want_bloom_filter for the Hash node. + * + * - ExecHashTableCreate (in nodeHash.c) builds the actual bloom_filter + * when HashState.want_bloom_filter is set (so no work happens when + * nobody will probe). + * + * - Nodes with non-NIL plan->bloom_filters (and supporting Bloom + * filters) call ExecInitBloomFilters() during its own ExecInit, + * which looks up the producer node (in the EState), compiles + * ExprStates for the hash expressions, etc. The filter state + * (BloomFilterState) gets added to ps->bloom_filters (a node may + * have multiple Bloom filters from different hash joins). + * + * - The per-tuple loop of the scan node calls ExecBloomFilters() (much + * like ExecQual) to test the tuple against every attached filter, + * dropping it on the first filter that excludes it. For scan nodes + * this call happens in ExecScanExtended. + * + * The scan nodes reach the Bloom filter via the HashJoinState pointer + * added to EState.es_bloom_producers, so that the rescans etc. (filter + * freed + recreated when the hash table is destroyed and rebuilt) are + * transparent to the consumer. The Bloom filter gets reallocated after + * a rescan, so the pointer to it may change. + * + * XXX It's possible the Bloom filter gets pushed down to a node that + * fails to initialize/use it. It'll be added to the bloom_filters list, + * but if the node does not call ExecInitBloomFilters, the filter will + * be unused. + * ---------------------------------------------------------------- + */ + +/* + * Lookup the HashJoinState for a producer by plan_node_id in the + * EState's es_bloom_producers list. Returns NULL if no matching + * producer has registered yet (which can happen for filters attached to + * recipients in trees where the producer hasn't ExecInit'd yet -- in + * normal execution we always register first). + */ +static HashJoinState * +LookupBloomFilterProducer(EState *estate, int bloom_filter_id) +{ + ListCell *lc; + + foreach(lc, estate->es_bloom_producers) + { + HashJoinState *hjstate = (HashJoinState *) lfirst(lc); + HashJoin *plan = (HashJoin *) hjstate->js.ps.plan; + + if (plan->bloom_filter_id == bloom_filter_id) + return hjstate; + } + return NULL; +} + +/* + * ExecBloomFilterHash + * Calculate the hash value for a tuple in the recipient node. + * + * Uses the per-key ExprStates initialized by ExecInitBloomFilters. Mirrors the + * scheme used by the Hash node, so that it matches what was inserted into the + * hashtable (and filter). + * + * Returns false if a strict key is NULL: such a tuple can't match anything in + * the bloom filter, but we still must let it pass through to the upstream join + * so the join (rather than us) decides what to do with it (e.g. emit + * NULL-extended for an outer join). + * + * XXX I'm not sure about this strict/NULL business. + */ +static inline bool +ExecBloomFilterHash(BloomFilterState *bfs, ExprContext *econtext, + uint32 *hashvalue) +{ + bool isnull; + uint32 hash; + + hash = DatumGetUInt32(ExecEvalExpr(bfs->keys, econtext, &isnull)); + + if (isnull) + return 0; /* XXX correct? do we care about NULL values? */ + + *hashvalue = hash; + return true; +} + +/* + * ADAPTIVE BEHAVIOR + * + * If the bloom filter lets through most (or all) tuples, it becomes somewhat + * useless - we're just wasting CPU cycles, getting nothing in return. We could + * simply stop using such filter. But we've already paid quite a bit to build + * it, and maybe the data set is not uniform and we'll get into a part where + * fewer tuples pass. + * + * So we're evaluating the match rate for windows of 1000 probes. If more than + * 90% match, we start sampling 1% of the probes (i.e. 99% it treated as a + * match without looking at the filter). And if the match rate drops below 80%, + * we stop the sampling and all probes go to the filter. + * + * XXX These are empirical values, picked based on experiments. "Perfect" + * values depend on hardware, number of keys, data types, ... and maybe even + * on how many hash joins / pushed-down filters there are, and how deep (the + * deeper the bigger the benefit). + * + * XXX Maybe we should sample more probes, or maybe the window should be a bit + * smaller? With 1% and 1000 probes per window, it'll take 100k probes to + * enable the filter again. That seems like a lot. + * + * XXX We should probably track the number of times we "disabled" the filter, + * and what fraction of entries were "let through" during sampling periods. + * + * XXX There's an intentional gap between low/high thresholds, to add a bit + * of hysteresis into the behavior, so it does not flap all the time. + */ +#define BLOOM_ADAPTIVE_WINDOW_SIZE 1000 +#define BLOOM_ADAPTIVE_HIGH_MATCH_PERCENT 90 +#define BLOOM_ADAPTIVE_LOW_MATCH_PERCENT 80 +#define BLOOM_ADAPTIVE_SAMPLE_RATE 100 + +/* + * ExecBloomFilterShouldProbe + * Decide if the next tuple should probe the bloom filter. + * + * Returns true if the next value should actually probe the bloom filter + * We sample 1/100 (1/BLOOM_ADAPTIVE_SAMPLE_RATE) probes, i.e. 1%. + */ +static inline bool +ExecBloomFilterShouldProbe(BloomFilterState *bfs) +{ + if (!bfs->adaptiveSampling) + return true; + + bfs->adaptiveSampleCounter++; + + if (bfs->adaptiveSampleCounter >= BLOOM_ADAPTIVE_SAMPLE_RATE) + { + bfs->adaptiveSampleCounter = 0; + return true; + } + + return false; +} + +/* + * ExecBloomFilterUpdateAdaptiveState + * Update the adaptive state for sampling the probes. + * + * Adjust the adaptive behavior every 1000 probes. If too many probes match, + * stop using the filter (and just sample 1% of probes instead). If we were + * sampling, and the fraction of matches drops enough, stop the sampling. + */ +static inline void +ExecBloomFilterUpdateAdaptiveState(BloomFilterState *bfs, bool match) +{ + bfs->adaptiveWindowProbes++; + if (match) + bfs->adaptiveWindowMatches++; + + /* have we done enough probes in this window? */ + if (bfs->adaptiveWindowProbes >= BLOOM_ADAPTIVE_WINDOW_SIZE) + { + uint64 match_percent; + + /* fraction of matches */ + match_percent = (bfs->adaptiveWindowMatches * 100) / + bfs->adaptiveWindowProbes; + + if (!bfs->adaptiveSampling && + match_percent > BLOOM_ADAPTIVE_HIGH_MATCH_PERCENT) + { + /* Too many matches - start sampling. */ + bfs->adaptiveSampling = true; + bfs->adaptiveSampleCounter = 0; + } + else if (bfs->adaptiveSampling && + match_percent < BLOOM_ADAPTIVE_LOW_MATCH_PERCENT) + { + /* Stop sampling if the match fraction got low enough. */ + bfs->adaptiveSampling = false; + bfs->adaptiveSampleCounter = 0; + } + + /* in any case, start a new window of probes */ + bfs->adaptiveWindowProbes = 0; + bfs->adaptiveWindowMatches = 0; + } +} + +/* + * ExecBloomFilters + * Probe bloom filters for the current slot. + * + * Test the slot in the expression context (set by the scan node) against + * every bloom filter attached to the node. Returns true if the tuple matches + * all filters (some of which may have been skipped, because the hash table + * isn't built yet); false if at least one filter conclusively excludes it. + * + * 'filters' is a list of BloomFilterState for each filter, pushed to the + * node (stored in planstate->bloom_filters). It may be NIL, which means + * are no filters, and the function simply returns NULL. + * + * The caller is responsible for having set econtext->ecxt_scantuple to + * 'slot' first. We do not reset the per-tuple context here (it's up to the + * scan node). + * + * Designed to be called like ExecQual from the recipient's per-tuple + * loop. See ExecScanExtended for the scan-node integration point. + * + * XXX We're pushing filters to scan nodes, which set the scan slot. And + * setrefs.c is currently wired to do fix_scan_bloom_filters, called from + * set_plan_refs. If we decide to push filters to other nodes (e.g. joins), + * this may need some rework. + */ +bool +ExecBloomFilters(List *filters, ExprContext *econtext) +{ + ListCell *lc; + + /* bail out if no filters */ + if (filters == NIL) + return true; + + foreach(lc, filters) + { + BloomFilterState *bfs = (BloomFilterState *) lfirst(lc); + HashJoinState *producer = bfs->producer; + HashState *hashNode; + bloom_filter *bf; + uint32 hashvalue; + + /* Producer should always exist (resolved at init time). */ + Assert(producer != NULL); + + /* + * The hashtable (and the bloom filter) is built lazily the first time + * it needs to do a lookup. Until then, assume everything matches + * everything through. Once the filter is in place, start probing it. + * + * XXX It should only take a couple tuples (maybe just a single one) + * from the scan node before the filter is available. + */ + hashNode = castNode(HashState, innerPlanState(&producer->js.ps)); + bf = hashNode->bloom_filter; + if (bf == NULL) + continue; + + /* + * When recent bloom probes mostly pass through, probe only a sample + * of values to avoid spending work on an ineffective filter. Sampled + * probes keep updating the recent match fraction, so filtering + * resumes for every value once the filter becomes selective again. + */ + if (!ExecBloomFilterShouldProbe(bfs)) + continue; + + /* NULL strict key: tuple cannot be in the filter, pass through. */ + if (!ExecBloomFilterHash(bfs, econtext, &hashvalue)) + continue; + + /* + * XXX It's a bit silly the counters are in two places. We should keep + * just the hashNode counters, and get rid of bfs counters. + */ + bfs->checked++; + hashNode->bloomFilterChecked++; + + /* If not matching, we're done - reject the tuple. */ + if (bloom_lacks_element(bf, + (unsigned char *) &hashvalue, + sizeof(hashvalue))) + { + bfs->rejected++; + hashNode->bloomFilterRejected++; + ExecBloomFilterUpdateAdaptiveState(bfs, false); + return false; + } + + ExecBloomFilterUpdateAdaptiveState(bfs, true); + } + + return true; +} + +/* + * ExecInitBloomFilters + * Initialize state for pushed-down bloom filters. + * + * Called by nodes that want to act as a recipient of pushed-down filters, + * after the node's projection / scan-tuple slot are set up, just like + * for regular quals. + * + * Walks the plan's bloom_filters list and produces a list of BloomFilterState + * nodes, stored in planstate->bloom_filters. The producer HashJoinState node + * is resolved here, once, via EState.es_bloom_producers; so that no lookup is + * needed at probe time (the bloom_filter pointer may change on rescan, but + * that's not what we store). + * + * 'output_slot' is the slot whose values the filter expressions will be + * evaluated against (i.e. the same slot the surrounding qual evaluates + * against, post-setrefs). + * + * XXX For now this has to be the scan slot. See the comment about setrefs + * a bit earlier. Could be relaxed later, if we support to pushdown to + * other node types. + * + * XXX The filter states are initialized in es_query_cxt, but maybe that's + * too long-lived. The states live only as long as the recipient node. + */ +void +ExecInitBloomFilters(PlanState *planstate, TupleTableSlot *output_slot) +{ + Plan *plan = planstate->plan; + EState *estate = planstate->state; + List *result = NIL; + ListCell *lc; + MemoryContext oldctx; + + /* bail out if there are no pushed-down filters */ + if (plan->bloom_filters == NIL) + return; + + oldctx = MemoryContextSwitchTo(estate->es_query_cxt); + + foreach(lc, plan->bloom_filters) + { + BloomFilter *bf = lfirst_node(BloomFilter, lc); + BloomFilterState *bfs; + int nkeys; + + nkeys = list_length(bf->filter_exprs); + Assert(nkeys > 0); + Assert(nkeys == list_length(bf->hashops)); + Assert(nkeys == list_length(bf->hashcollations)); + + bfs = makeNode(BloomFilterState); + + /* XXX some of this is redundant */ + bfs->filter = bf; + bfs->producer_id = bf->producer_id; + bfs->producer = LookupBloomFilterProducer(estate, bf->producer_id); + + /* initialize the expression state for the hashvalue calculation */ + { + Oid *outer_hashfuncid = palloc_array(Oid, nkeys); + Oid *inner_hashfuncid = palloc_array(Oid, nkeys); + bool *hash_strict = palloc_array(bool, nkeys); + ListCell *lc2; + + /* + * Determine the hash function for each side of the join for the + * given join operator, and detect whether the join operator is + * strict. + */ + foreach(lc2, bf->hashops) + { + Oid hashop = lfirst_oid(lc2); + int i = foreach_current_index(lc2); + + if (!get_op_hash_functions(hashop, + &outer_hashfuncid[i], + &inner_hashfuncid[i])) + elog(ERROR, + "could not find hash function for hash operator %u", + hashop); + hash_strict[i] = op_strict(hashop); + } + + /* state for the hash value calculation */ + bfs->keys = ExecBuildHash32Expr(output_slot->tts_tupleDescriptor, + output_slot->tts_ops, + outer_hashfuncid, + bf->hashcollations, + bf->filter_exprs, + hash_strict, + planstate, + 0); + } + + result = lappend(result, bfs); + } + + planstate->bloom_filters = result; + + MemoryContextSwitchTo(oldctx); +} + +/* + * ExecRegisterBloomFilterProducer + * Register the pushed downn bloom filter. + * + * Called by ExecInitHashJoin (before recursing into the outer subtree) + * to register this HashJoinState as a producer for the pushed-down filter. + * Recipients in the outer subtree will look us up here by plan_node_id. + */ +void +ExecRegisterBloomFilterProducer(HashJoinState *hjstate) +{ + EState *estate = hjstate->js.ps.state; + + estate->es_bloom_producers = lappend(estate->es_bloom_producers, hjstate); +} diff --git a/src/backend/executor/nodeIndexonlyscan.c b/src/backend/executor/nodeIndexonlyscan.c index 8afcbde6b97..e632cc773c9 100644 --- a/src/backend/executor/nodeIndexonlyscan.c +++ b/src/backend/executor/nodeIndexonlyscan.c @@ -605,6 +605,9 @@ ExecInitIndexOnlyScan(IndexOnlyScan *node, EState *estate, int eflags) indexstate->recheckqual = ExecInitQual(node->recheckqual, (PlanState *) indexstate); + ExecInitBloomFilters((PlanState *) indexstate, + indexstate->ss.ss_ScanTupleSlot); + /* * If we are just doing EXPLAIN (ie, aren't going to run the plan), stop * here. This allows an index-advisor plugin to EXPLAIN a plan containing diff --git a/src/backend/executor/nodeIndexscan.c b/src/backend/executor/nodeIndexscan.c index 39f6691ee35..ef56522fbc5 100644 --- a/src/backend/executor/nodeIndexscan.c +++ b/src/backend/executor/nodeIndexscan.c @@ -970,6 +970,9 @@ ExecInitIndexScan(IndexScan *node, EState *estate, int eflags) indexstate->indexorderbyorig = ExecInitExprList(node->indexorderbyorig, (PlanState *) indexstate); + ExecInitBloomFilters((PlanState *) indexstate, + indexstate->ss.ss_ScanTupleSlot); + /* * If we are just doing EXPLAIN (ie, aren't going to run the plan), stop * here. This allows an index-advisor plugin to EXPLAIN a plan containing diff --git a/src/backend/executor/nodeSamplescan.c b/src/backend/executor/nodeSamplescan.c index f3d273e1c5e..738502433b4 100644 --- a/src/backend/executor/nodeSamplescan.c +++ b/src/backend/executor/nodeSamplescan.c @@ -147,6 +147,9 @@ ExecInitSampleScan(SampleScan *node, EState *estate, int eflags) scanstate->repeatable = ExecInitExpr(tsc->repeatable, (PlanState *) scanstate); + ExecInitBloomFilters((PlanState *) scanstate, + scanstate->ss.ss_ScanTupleSlot); + /* * If we don't have a REPEATABLE clause, select a random seed. We want to * do this just once, since the seed shouldn't change over rescans. diff --git a/src/backend/executor/nodeSeqscan.c b/src/backend/executor/nodeSeqscan.c index b8c528ca089..88a6f6622bd 100644 --- a/src/backend/executor/nodeSeqscan.c +++ b/src/backend/executor/nodeSeqscan.c @@ -268,6 +268,9 @@ ExecInitSeqScan(SeqScan *node, EState *estate, int eflags) scanstate->ss.ps.qual = ExecInitQual(node->scan.plan.qual, (PlanState *) scanstate); + ExecInitBloomFilters((PlanState *) scanstate, + scanstate->ss.ss_ScanTupleSlot); + /* * When EvalPlanQual() is not in use, assign ExecProcNode for this node * based on the presence of qual and projection. Each ExecSeqScan*() diff --git a/src/backend/executor/nodeTidrangescan.c b/src/backend/executor/nodeTidrangescan.c index b387ed6c308..b00a3736b99 100644 --- a/src/backend/executor/nodeTidrangescan.c +++ b/src/backend/executor/nodeTidrangescan.c @@ -434,6 +434,9 @@ ExecInitTidRangeScan(TidRangeScan *node, EState *estate, int eflags) tidrangestate->ss.ps.qual = ExecInitQual(node->scan.plan.qual, (PlanState *) tidrangestate); + ExecInitBloomFilters((PlanState *) tidrangestate, + tidrangestate->ss.ss_ScanTupleSlot); + TidExprListCreate(tidrangestate); /* diff --git a/src/backend/executor/nodeTidscan.c b/src/backend/executor/nodeTidscan.c index 6641df10999..f7ba78a63fc 100644 --- a/src/backend/executor/nodeTidscan.c +++ b/src/backend/executor/nodeTidscan.c @@ -551,6 +551,9 @@ ExecInitTidScan(TidScan *node, EState *estate, int eflags) tidstate->ss.ps.qual = ExecInitQual(node->scan.plan.qual, (PlanState *) tidstate); + ExecInitBloomFilters((PlanState *) tidstate, + tidstate->ss.ss_ScanTupleSlot); + TidExprListCreate(tidstate); /* diff --git a/src/backend/lib/bloomfilter.c b/src/backend/lib/bloomfilter.c index 73b3768a172..fbcf788e271 100644 --- a/src/backend/lib/bloomfilter.c +++ b/src/backend/lib/bloomfilter.c @@ -192,6 +192,25 @@ bloom_prop_bits_set(bloom_filter *filter) return bits_set / (double) filter->m; } +/* + * Total bitset size, in bits. Useful for EXPLAIN instrumentation: + * divide by 8 to get the bitset's memory footprint in bytes. + */ +uint64 +bloom_total_bits(bloom_filter *filter) +{ + return filter->m; +} + +/* + * Number of hash functions in use. + */ +int +bloom_hash_funcs(bloom_filter *filter) +{ + return filter->k_hash_funcs; +} + /* * Which element in the sequence of powers of two is less than or equal to * target_bitset_bits? diff --git a/src/backend/optimizer/path/allpaths.c b/src/backend/optimizer/path/allpaths.c index 9c040ecefc8..6e750e69172 100644 --- a/src/backend/optimizer/path/allpaths.c +++ b/src/backend/optimizer/path/allpaths.c @@ -107,6 +107,9 @@ static void set_rel_consider_parallel(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte); static void set_plain_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte); +static List *find_interesting_bloom_filters(PlannerInfo *root, + RelOptInfo *rel); +static void generate_expected_filter_paths(PlannerInfo *root, RelOptInfo *rel); static void set_tablesample_rel_size(PlannerInfo *root, RelOptInfo *rel, RangeTblEntry *rte); static void set_tablesample_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, @@ -607,6 +610,16 @@ set_rel_pathlist(PlannerInfo *root, RelOptInfo *rel, !bms_equal(rel->relids, root->all_query_rels)) generate_useful_gather_paths(root, rel, false); + /* + * For plain base relations, consider generating additional scan paths + * that anticipate a Bloom filter being pushed down from a hash join above + * (see find_interesting_bloom_filters). These paths have reduced row + * estimates and are consumed by join path generation. + */ + if (rel->reloptkind == RELOPT_BASEREL && + rte->rtekind == RTE_RELATION) + generate_expected_filter_paths(root, rel); + /* Now find the cheapest of the paths for this rel */ set_cheapest(rel); @@ -888,6 +901,463 @@ create_plain_partial_paths(PlannerInfo *root, RelOptInfo *rel) add_partial_path(rel, create_seqscan_path(root, rel, NULL, parallel_workers)); } +/* + * bloom_em_matches_anybarevar + * ec_matches_callback used by find_interesting_bloom_filters: accept any + * EquivalenceClass member that is a bare Var of the target relation. The + * Bloom filter pushdown logic (createplan.c) only supports recipients whose + * join keys are bare Vars, so we mirror that requirement here. + */ +static bool +bloom_em_matches_anybarevar(PlannerInfo *root, RelOptInfo *rel, + EquivalenceClass *ec, EquivalenceMember *em, + void *arg) +{ + Var *var; + + /* We're looking only for bare Var expressions. */ + if (!IsA(em->em_expr, Var)) + return false; + + /* + * Is the Var referencing a normal (non-system) attribute in the relation + * we're processing (generating scans for)? + * + * FIXME Can we have (varlevelsup != 0) for baserels? I don't think we can + * have outer referecenses in that place. + */ + var = (Var *) em->em_expr; + if (var->varno != rel->relid || + var->varattno <= 0 || + var->varlevelsup != 0) + return false; + + return true; +} + +/* + * bloom_filter_recipient_reachable + * Check that a Bloom filter owned by owner_relid and built from + * build_relids could actually be pushed to the owner's scan. + * + * A pushed-down filter removes owner tuples that have no match on the build + * side, so it can only be applied where the join that realizes it drops such + * unmatched owner (probe) tuples. If an outer join null-extends the owner + * before it can be joined to the build side, the filter would change the + * result and is therefore unusable: at plan time find_bloom_filter_recipient + * would refuse to descend into that side and find no recipient (see also + * bloom_join_side_preserved, which is checking for this situation). + * + * Picking such a filter only to throw it away later wastes planner effort, + * and we might also ignore some other filters because of that. It's better + * to eliminate it right away. + * + * This is primarily an optimization - we don't want to generate paths that + * would ultimately be useless, and possibly not generating paths for other + * filters. The correctness is still guaranteed by the propagation logic in + * compute_join_expected_filters(), which rejects cases that would carry a + * filter across a non-preserved join side. That guarantees we don't pick a + * plan with such filters, only to find about the issue in createplan.c. + */ +static bool +bloom_filter_recipient_reachable(PlannerInfo *root, Index owner_relid, + Relids build_relids) +{ + ListCell *lc; + + foreach(lc, root->join_info_list) + { + SpecialJoinInfo *sjinfo = (SpecialJoinInfo *) lfirst(lc); + + switch (sjinfo->jointype) + { + case JOIN_LEFT: + case JOIN_ANTI: + + /* + * The syntactic RHS is null-extended. If the owner is on it + * but the build side is reached from the preserved LHS, the + * owner must cross this outer join on its nullable side. + */ + if (bms_is_member(owner_relid, sjinfo->syn_righthand) && + !bms_is_subset(build_relids, sjinfo->syn_righthand)) + return false; + break; + case JOIN_FULL: + + /* + * Both sides are null-extended, so the filter is unusable + * whenever the owner and the build side sit on opposite sides + * of the join. + */ + if (bms_is_member(owner_relid, sjinfo->syn_lefthand) && + !bms_is_subset(build_relids, sjinfo->syn_lefthand)) + return false; + if (bms_is_member(owner_relid, sjinfo->syn_righthand) && + !bms_is_subset(build_relids, sjinfo->syn_righthand)) + return false; + break; + default: + /* INNER and SEMI joins never null-extend the owner. */ + break; + } + } + + return true; +} + +/* + * find_interesting_bloom_filters + * Identify Bloom filters that a hash join above this scan could push down, + * and that are selective enough to be worth costing for. + * + * We look for hashjoinable equality join clauses where this rel's side is a + * bare Var, derived both from EquivalenceClasses and from non-EC joininfo + * clauses. Clauses are grouped by the relids on the other ("build") side of + * the join; each group becomes a candidate filter whose expected surviving + * fraction is estimated as the semijoin selectivity of those clauses. + * + * A candidate is "interesting" only if it is expected to eliminate at least + * bloom_filter_pushdown_threshold of the rel's tuples. We keep at most + * bloom_filter_pushdown_max of the most selective candidates, and return them + * as a list of ExpectedFilter nodes. + * + * XXX This needs to be careful to not interfere with the general selectivity + * estimation, performed by clauselist_selectivity(). We'll estimate the filter + * selectivity using a made-up sjinfo with JOIN_INNER, which may not match + * the actual join. The selectivities must not leak - this is why this function + * does not collect the RestrictInfos but only the clauses. If we used the + * RestrictInfos, the clauselist_selectivity would cache the incorrect result + * in them, and it'd affect the planning in weird ways. + * + * FIXME Maybe there's a better way to calculate the filter selectivity? + */ +static List * +find_interesting_bloom_filters(PlannerInfo *root, RelOptInfo *rel) +{ + List *candidates; + List *group_relids = NIL; /* parallel: Relids per group */ + List *group_clauses = NIL; /* parallel: List of clauses */ + List *result = NIL; + ListCell *lc; + + if (!enable_hashjoin_bloom) + return NIL; + + if (bloom_filter_pushdown_max <= 0) + return NIL; + + if (rel->reloptkind != RELOPT_BASEREL) + return NIL; + + /* Collect candidate hashjoinable equality clauses for this rel. */ + candidates = generate_implied_equalities_for_all_columns(root, rel, + bloom_em_matches_anybarevar, + NULL, NULL); + + foreach(lc, rel->joininfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(lc); + + /* EC-derived clauses are already covered above. */ + if (rinfo->parent_ec != NULL) + continue; + + candidates = lappend(candidates, rinfo->clause); + } + + /* Group candidate clauses by their build-side relids. */ + foreach(lc, candidates) + { + Node *clause = (Node *) lfirst(lc); + Node *left; + Node *right; + Node *ownerexpr; + Oid opno; + Relids clause_relids; + Relids build_relids; + int buildrel; + ListCell *lc2; + ListCell *lc3; + bool found; + + /* strip RestrictInfo (see comment above) */ + if (IsA(clause, RestrictInfo)) + clause = (Node *) ((RestrictInfo *) clause)->clause; + + /* + * Only care about (Expr op Expr) clauses. We know one side has to be + * a bare Var node, from the "owner" side (which is the scan node). + * The other side can be arbitrary expression on the other relation. + */ + if (!is_opclause(clause) || + list_length(((OpExpr *) clause)->args) != 2) + continue; + + opno = ((OpExpr *) clause)->opno; + left = get_leftop(clause); + right = get_rightop(clause); + + /* Identify which side is a bare Var of this rel (the owner side). */ + + /* + * XXX replace this with a macro shared with + * bloom_em_matches_anybarevar + */ + if (IsA(left, Var) && ((Var *) left)->varno == rel->relid && + ((Var *) left)->varattno > 0 && ((Var *) left)->varlevelsup == 0) + ownerexpr = left; + else if (IsA(right, Var) && ((Var *) right)->varno == rel->relid && + ((Var *) right)->varattno > 0 && + ((Var *) right)->varlevelsup == 0) + ownerexpr = right; + else + continue; + + /* Operator must be hashjoinable on the owner's input type. */ + if (!op_hashjoinable(opno, exprType(ownerexpr))) + continue; + + /* + * The build side must be a single base relation; that's what the + * recipient lookup and our selectivity estimate can handle. + * + * XXX I don't think this restriction is necessary. We can allow the + * build side to be a join. I don't see why that would be a problem. + */ + clause_relids = pull_varnos(root, (Node *) clause); + build_relids = bms_difference(clause_relids, rel->relids); + if (!bms_get_singleton_member(build_relids, &buildrel) || + buildrel >= root->simple_rel_array_size || + root->simple_rel_array[buildrel] == NULL || + root->simple_rel_array[buildrel]->reloptkind != RELOPT_BASEREL) + { + bms_free(build_relids); + continue; + } + + /* Add to an existing group, or start a new one. */ + + /* + * XXX Maybe we sould have a HTAB with the relids as a key? But the + * lists should not be that long, I think. + */ + found = false; + forboth(lc2, group_relids, lc3, group_clauses) + { + Relids grelids = (Relids) lfirst(lc2); + + if (bms_equal(grelids, build_relids)) + { + lfirst(lc3) = lappend((List *) lfirst(lc3), clause); + found = true; + + /* added to an existing group, don't keep the relids around */ + bms_free(build_relids); + + break; + } + } + + if (!found) /* new group */ + { + group_relids = lappend(group_relids, build_relids); + group_clauses = lappend(group_clauses, list_make1(clause)); + } + } + + /* + * We have collected all potentially intresting filters. Evaluate + * selectivity of each group and keep only the most interesting filters. + * Filters have to eliminate at least bloom_filter_pushdown_threshold + * tuples, and we keep only bloom_filter_pushdown_max most selective ones. + */ + { + ListCell *lcr = list_head(group_relids); + ListCell *lcc = list_head(group_clauses); + + while (lcr != NULL && lcc != NULL) + { + Relids build_relids = (Relids) lfirst(lcr); + List *clauses = (List *) lfirst(lcc); + SpecialJoinInfo sjinfo; + Selectivity sel; + + init_dummy_sjinfo(&sjinfo, rel->relids, build_relids); + sjinfo.jointype = JOIN_SEMI; + + sel = clauselist_selectivity(root, clauses, 0, JOIN_SEMI, &sjinfo); + + if ((sel <= 1.0 - bloom_filter_pushdown_threshold) && + (sel > 0.0) && /* XXX seems unnecessary */ + bloom_filter_recipient_reachable(root, rel->relid, build_relids)) + { + ExpectedFilter *f = makeNode(ExpectedFilter); + + f->owner_relid = rel->relid; + f->build_relids = build_relids; + f->clauses = clauses; + f->selectivity = sel; + result = lappend(result, f); + } + + lcr = lnext(group_relids, lcr); + lcc = lnext(group_clauses, lcc); + } + } + + /* + * We only connsider a limited number of interesting filters, to prevent + * path explosion. If we found too many, keep only the most selective ones + * (with smallest surviving fraction of tuples), to bound the number of + * generated paths. + * + * XXX This also aligns with good join orders - those tend to perform the + * most selective joins first. So we get to build the filters soon, even + * if the hashjoin optimization is not disabled. + */ + while (list_length(result) > bloom_filter_pushdown_max) + { + ExpectedFilter *worst = NULL; + ListCell *lcw; + + foreach(lcw, result) + { + ExpectedFilter *f = (ExpectedFilter *) lfirst(lcw); + + if (worst == NULL || f->selectivity > worst->selectivity) + worst = f; + } + result = list_delete_ptr(result, worst); + } + + return result; +} + +/* + * generate_expected_filter_paths + * Generate additional scan paths that anticipate one or more pushed-down + * Bloom filters. + * + * For each non-empty subset of the interesting filters, we clone every eligible + * existing scan path, reducing its row estimate by the combined selectivity and + * attaching the corresponding ExpectedFilter nodes. + * + * These paths are kept alongside the regular paths (add_path keeps paths with + * differing expected_filters) and are consumed by join path generation; + * set_cheapest never selects them. + * + * XXX We must not clone paths that already have expected filters. + * + * XXX The cloning is a rather dirty way to copy paths. It does not readjust the + * cost in a reasonable way. For example custom scans could do something smart + * with the filters, so it should have a chance to deal with that. A cleaner + * solution might be to actually pass the filters to the various "create" + * function, like create_seqscan_path/... For CustomScan nodes we can probably + * do most of this in the set_rel_pathlist_hook, somewhere. Maybe that needs + * some helper methods, though. And maybe it will need to pass some of the info + * through the callbacks? Not sure, someone has to try that. + * + * XXX This may need some major changes to work with custom scans. Right now we + * only consider filters exactly matching the hash keys, so if the hashjoin is + * on (t1.a = t2.a AND t1.b = t2.b), then the filter will be on (a,b). But a + * custom scan may prefer "split" filters on each column independently. We'd + * need a way for the custom scan to indicate that, and we'd need to apply this + * only to the "matching" scan paths (and not to any other scan paths). But + * we only look at the paths after selecting the "interesting" filters, so we'd + * need to rethink that - we'd need to make the "interesting" filters specific + * to a path, or something like that. + */ +static void +generate_expected_filter_paths(PlannerInfo *root, RelOptInfo *rel) +{ + List *filters; + List *basepaths = NIL; + int nfilters; + uint32 combo; + ListCell *lc; + + filters = find_interesting_bloom_filters(root, rel); + if (filters == NIL) + return; + + nfilters = list_length(filters); + + /* + * Snapshot the existing unparameterized, non-partial scan paths of a + * supported type. We must snapshot before calling add_path(), which + * mutates rel->pathlist. + */ + foreach(lc, rel->pathlist) + { + Path *path = (Path *) lfirst(lc); + + /* XXX Is parameterization really a problem? Always? */ + if (path->param_info != NULL || path->expected_filters != NIL) + continue; + + switch (nodeTag(path)) + { + case T_Path: + case T_IndexPath: + case T_BitmapHeapPath: + case T_TidPath: + case T_TidRangePath: + basepaths = lappend(basepaths, path); + break; + default: + break; + } + } + + if (basepaths == NIL) + return; + + /* + * Generate all combinations of the interesting filters. We do that by + * iterating 1 to (2^n-1), which generates all bitmask in between. Those + * are the subsets. + * + * XXX This is a good demonstration why we need to keep the number of + * filters low + * + * XXX Maybe we should also stop adding filters once the other filters + * already eliminate enought tuples. Say, we know F1 alone eliminates 99% + * tuples. Does it make sense to also consider [F1,F2]? Probably not. We + * could track "maximum" sets, and reject combinations containing one of + * those. We'd need to generate sets of increasing size, the iteration + * does not do that. But that's not hard. + */ + for (combo = 1; combo < ((uint32) 1 << nfilters); combo++) + { + List *subset = NIL; + int i = 0; + ListCell *lcf; + + foreach(lcf, filters) + { + if (combo & ((uint32) 1 << i)) + subset = lappend(subset, lfirst(lcf)); + i++; + } + + /* + * All filtered paths for this combo share the same expected_filters + * list. That's safe: the list is never modified, and add_path() only + * ever frees the Path node itself, not its expected_filters. + */ + foreach(lc, basepaths) + { + Path *base = (Path *) lfirst(lc); + Path *newpath; + + newpath = create_filtered_scan_path(root, base, subset); + if (newpath != NULL) + add_path(rel, newpath); + } + } +} + /* * set_tablesample_rel_size * Set size estimates for a sampled relation diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index ac523ecf9a8..a29e63e8738 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -157,6 +157,7 @@ bool enable_material = true; bool enable_memoize = true; bool enable_mergejoin = true; bool enable_hashjoin = true; +bool enable_hashjoin_bloom = true; bool enable_gathermerge = true; bool enable_partitionwise_join = false; bool enable_partitionwise_aggregate = false; @@ -166,6 +167,23 @@ bool enable_partition_pruning = true; bool enable_presorted_aggregate = true; bool enable_async_append = true; +/* + * Minimum fraction of outer tuples a pushed-down hash-join Bloom filter must + * be expected to eliminate for the planner to treat it as "interesting" and + * generate filter-aware scan paths. A value of 0.3 means a filter is only + * considered if it is expected to discard at least 30% of the scanned tuples. + */ +double bloom_filter_pushdown_threshold = 0.3; + +/* + * Upper bound on the number of distinct interesting Bloom filters considered + * for a single scan relation. This bounds the number of additional paths + * generated per scan (the planner enumerates non-empty subsets of the + * interesting filters, i.e. up to 2^bloom_filter_pushdown_max - 1 extra + * paths per base scan path). + */ +int bloom_filter_pushdown_max = 3; + typedef struct { PlannerInfo *root; diff --git a/src/backend/optimizer/path/equivclass.c b/src/backend/optimizer/path/equivclass.c index e3697df51a2..ae7311053a4 100644 --- a/src/backend/optimizer/path/equivclass.c +++ b/src/backend/optimizer/path/equivclass.c @@ -75,6 +75,15 @@ static RestrictInfo *create_join_clause(PlannerInfo *root, EquivalenceMember *leftem, EquivalenceMember *rightem, EquivalenceClass *parent_ec); +static int generate_implied_equalities_for_column_ec(PlannerInfo *root, + RelOptInfo *rel, + EquivalenceClass *cur_ec, + ec_matches_callback_type callback, + void *callback_arg, + Relids prohibited_rels, + Relids parent_relids, + bool is_child_rel, + List **result); static bool reconsider_outer_join_clause(PlannerInfo *root, OuterJoinClauseInfo *ojcinfo, bool outer_on_left); @@ -3211,6 +3220,107 @@ eclass_member_iterator_next(EquivalenceMemberIterator *it) return NULL; } +/* + * generate_implied_equalities_for_column_ec + * Workhorse for generate_implied_equalities_for_column() and + * generate_implied_equalities_for_all_columns(). Considers a single + * EquivalenceClass cur_ec: if it has a member matching the target column + * (as identified by the callback), generate EC-derived joinclauses + * equating that member to each other-relation member, appending them to + * *result. Returns the number of clauses generated. + */ +static int +generate_implied_equalities_for_column_ec(PlannerInfo *root, + RelOptInfo *rel, + EquivalenceClass *cur_ec, + ec_matches_callback_type callback, + void *callback_arg, + Relids prohibited_rels, + Relids parent_relids, + bool is_child_rel, + List **result) +{ + EquivalenceMemberIterator it; + EquivalenceMember *cur_em; + ListCell *lc2; + int ngenerated = 0; + + /* + * Won't generate joinclauses if const or single-member (the latter test + * covers the volatile case too) + */ + if (cur_ec->ec_has_const || list_length(cur_ec->ec_members) <= 1) + return 0; + + /* + * Scan members, looking for a match to the target column. Note that + * child EC members are considered, but only when they belong to the + * target relation. (Unlike regular members, the same expression could be + * a child member of more than one EC. Therefore, it's potentially + * order-dependent which EC a child relation's target column gets matched + * to. This is annoying but it only happens in corner cases, so for now + * we live with just reporting the first match. See also + * get_eclass_for_sort_expr.) + */ + setup_eclass_member_iterator(&it, cur_ec, rel->relids); + while ((cur_em = eclass_member_iterator_next(&it)) != NULL) + { + if (bms_equal(cur_em->em_relids, rel->relids) && + callback(root, rel, cur_ec, cur_em, callback_arg)) + break; + } + + if (!cur_em) + return 0; + + /* + * Found our match. Scan the other EC members and attempt to generate + * joinclauses. Ignore children here. + */ + foreach(lc2, cur_ec->ec_members) + { + EquivalenceMember *other_em = (EquivalenceMember *) lfirst(lc2); + Oid eq_op; + RestrictInfo *rinfo; + + /* Child members should not exist in ec_members */ + Assert(!other_em->em_is_child); + + /* Make sure it'll be a join to a different rel */ + if (other_em == cur_em || + bms_overlap(other_em->em_relids, rel->relids)) + continue; + + /* Forget it if caller doesn't want joins to this rel */ + if (bms_overlap(other_em->em_relids, prohibited_rels)) + continue; + + /* + * Also, if this is a child rel, avoid generating a useless join to + * its parent rel(s). + */ + if (is_child_rel && + bms_overlap(parent_relids, other_em->em_relids)) + continue; + + eq_op = select_equality_operator(cur_ec, + cur_em->em_datatype, + other_em->em_datatype); + if (!OidIsValid(eq_op)) + continue; + + /* set parent_ec to mark as redundant with other joinclauses */ + rinfo = create_join_clause(root, cur_ec, eq_op, + cur_em, other_em, + cur_ec); + + *result = lappend(*result, rinfo); + ngenerated++; + } + + return ngenerated; +} + /* * generate_implied_equalities_for_column * Create EC-derived joinclauses usable with a specific column. @@ -3233,6 +3343,10 @@ eclass_member_iterator_next(EquivalenceMemberIterator *it) * * The caller can pass a Relids set of rels we aren't interested in joining * to, so as to save the work of creating useless clauses. + * + * XXX This could reuse generate_implied_equalities_for_column_ec for the + * inner loop, similarly to generate_implied_equalities_for_all_columns, but I + * chose to not do that for now. Better keep this as is. */ List * generate_implied_equalities_for_column(PlannerInfo *root, @@ -3353,6 +3467,71 @@ generate_implied_equalities_for_column(PlannerInfo *root, return result; } +/* + * generate_implied_equalities_for_all_columns + * Like generate_implied_equalities_for_column, but returns EC-derived + * joinclauses for *every* column of the relation, rather than stopping at + * the first column (EquivalenceClass) that yields any clauses. + * + * generate_implied_equalities_for_column() is designed for parameterized-path + * generation, where the goal is to find a single usable joinclause per column + * and there is no value in returning clauses for more than one column at a + * time. Some callers, however, are interested in joinclauses on all of the + * relation's columns simultaneously (for example, the Bloom filter pushdown + * logic, which may push down filters derived from several different columns at + * once). This variant therefore visits all of the relation's + * EquivalenceClasses and accumulates clauses from each. + * + * As with generate_implied_equalities_for_column(), the result for any single + * column is a redundant set of clauses equating that column to each of the + * other-relation values it is known to be equal to. + * + * XXX We don't really need the last two arguments, but we keep this as close + * to generate_implied_equalities_for_column as possible. + */ +List * +generate_implied_equalities_for_all_columns(PlannerInfo *root, + RelOptInfo *rel, + ec_matches_callback_type callback, + void *callback_arg, + Relids prohibited_rels) +{ + List *result = NIL; + bool is_child_rel = (rel->reloptkind == RELOPT_OTHER_MEMBER_REL); + Relids parent_relids; + int i; + + /* Should be OK to rely on eclass_indexes */ + Assert(root->ec_merging_done); + + /* Indexes are available only on base or "other" member relations. */ + Assert(IS_SIMPLE_REL(rel)); + + /* If it's a child rel, we'll need to know what its parent(s) are */ + if (is_child_rel) + parent_relids = find_childrel_parents(root, rel); + else + parent_relids = NULL; /* not used, but keep compiler quiet */ + + i = -1; + while ((i = bms_next_member(rel->eclass_indexes, i)) >= 0) + { + EquivalenceClass *cur_ec = (EquivalenceClass *) list_nth(root->eq_classes, i); + + /* Sanity check eclass_indexes only contain ECs for rel */ + Assert(is_child_rel || bms_is_subset(rel->relids, cur_ec->ec_relids)); + + (void) generate_implied_equalities_for_column_ec(root, rel, cur_ec, + callback, callback_arg, + prohibited_rels, + parent_relids, + is_child_rel, + &result); + } + + return result; +} + /* * have_relevant_eclass_joinclause * Detect whether there is an EquivalenceClass that could produce diff --git a/src/backend/optimizer/path/joinpath.c b/src/backend/optimizer/path/joinpath.c index 713283a73aa..28e61807e7c 100644 --- a/src/backend/optimizer/path/joinpath.c +++ b/src/backend/optimizer/path/joinpath.c @@ -869,6 +869,271 @@ get_memoize_path(PlannerInfo *root, RelOptInfo *innerrel, return NULL; } +/* + * bloom_join_side_preserved + * Can a pushed-down Bloom filter be applied below the given side of a + * join of this type without changing the join's result? + * + * A Bloom filter removes tuples from the scan it is pushed to. That is only + * safe on a side whose unmatched tuples the join would drop anyway: dropping + * a tuple early then matches the join's behaviour. On a null-extended + * (preserved-other-side) input it is unsafe, because removing a tuple there + * could suppress, or spuriously emit, null-extended rows. + * + * This is the path-time counterpart of find_bloom_filter_recipient() in + * createplan.c, which descends a plan tree toward the recipient scan: it may + * only descend into a join's outer (resp. inner) child when this function + * returns true for to_outer = true (resp. false). Keeping the two in sync is + * what guarantees that every filter we realize at path time has a reachable + * recipient at plan time. + */ +bool +bloom_join_side_preserved(JoinType jointype, bool to_outer) +{ + switch (jointype) + { + case JOIN_INNER: + return true; + case JOIN_LEFT: + case JOIN_SEMI: + case JOIN_ANTI: + return to_outer; + case JOIN_RIGHT: + case JOIN_RIGHT_SEMI: + case JOIN_RIGHT_ANTI: + return !to_outer; + case JOIN_FULL: + return false; + default: + return false; + } +} + +/* + * jointype_realizes_bloom_filter + * Can a hash join of this type build and push a Bloom filter to its + * outer (probe) side? + * + * This must match the join-type check in try_push_bloom_filter(), so that a + * filter we cost for is actually realized at plan time. Only join types that + * drop unmatched outer (probe) tuples are safe, since the filter eliminates + * probe tuples lacking an inner match. + * + * Note JOIN_RIGHT qualifies: it preserves unmatched tuples of the inner (build) + * side, not the outer (probe) side, so dropping unmatched probe tuples is still + * correct. + */ +static bool +jointype_realizes_bloom_filter(JoinType jointype) +{ + switch (jointype) + { + case JOIN_INNER: + case JOIN_RIGHT: + case JOIN_SEMI: + case JOIN_RIGHT_SEMI: + case JOIN_RIGHT_ANTI: + return true; + default: + return false; + } +} + +/* + * hashjoin_pushes_filter_to + * Would create_hashjoin_plan() be able to push a Bloom filter built from + * these hash clauses down to the scan of owner_relid? + * + * This mirrors try_push_bloom_filter()'s requirement that every hash key on + * the outer side be a bare Var of a single base relation. 'outer_relids' is + * the set of relids on the outer (probe) side of the join. + */ +static bool +hashjoin_pushes_filter_to(List *hashclauses, Relids outer_relids, + Index owner_relid) +{ + ListCell *lc; + + if (hashclauses == NIL) + return false; + + foreach(lc, hashclauses) + { + RestrictInfo *ri = (RestrictInfo *) lfirst(lc); + Node *outerside; + Var *var; + + if (!is_opclause(ri->clause) || + list_length(((OpExpr *) ri->clause)->args) != 2) + return false; + + /* Pick the side that belongs to the outer relation. */ + if (bms_is_subset(ri->left_relids, outer_relids)) + outerside = get_leftop(ri->clause); + else if (bms_is_subset(ri->right_relids, outer_relids)) + outerside = get_rightop(ri->clause); + else + return false; + + if (!IsA(outerside, Var)) + return false; + var = (Var *) outerside; + if (var->varno != owner_relid || + var->varattno <= 0 || + var->varlevelsup != 0) + return false; + } + + return true; +} + +/* + * compute_join_expected_filters + * Determine the expected Bloom filters a prospective join path should + * carry, applying the propagation and contradiction rules. + * + * Each filter expected by an input path is either: + * + * - propagated upward (its build side is not yet fully joined in), or + * + * - realized by this join (a hash join that is the filter's source and can + * push the filter to its outer side): it is dropped from the result, since + * it has now been applied. When 'realized' is non-NULL, such filters are + * appended to *realized so the caller can record them on the join path and + * later propagate the pushdown decision to the plan, or + * + * - contradicted: the join is the filter's source but cannot push it (it is + * not a suitable hash join). In that case the input path cannot be used + * for this join, so *contradicted is set true and NIL is returned. + * + * 'is_hashjoin'/'hashclauses' describe the join method; hashclauses is only + * meaningful for hash joins. + */ +static List * +compute_join_expected_filters(PlannerInfo *root, + Path *outer_path, Path *inner_path, + JoinType jointype, bool is_hashjoin, + List *hashclauses, bool *contradicted, + List **realized) +{ + List *result = NIL; + Relids outer_relids = outer_path->parent->relids; + Relids inner_relids = inner_path->parent->relids; + Relids join_relids; + int pass; + + *contradicted = false; + if (realized != NULL) + *realized = NIL; + + /* Fast path: neither input expects any filter. */ + if (outer_path->expected_filters == NIL && + inner_path->expected_filters == NIL) + return NIL; + + join_relids = bms_union(outer_relids, inner_relids); + + /* Examine the filters from both inputs. */ + for (pass = 0; pass < 2; pass++) + { + List *filters = (pass == 0) ? outer_path->expected_filters + : inner_path->expected_filters; + ListCell *lc; + + foreach(lc, filters) + { + ExpectedFilter *f = (ExpectedFilter *) lfirst(lc); + bool owner_in_outer; + Relids owner_relids; + Relids other_relids; + + owner_in_outer = bms_is_member(f->owner_relid, outer_relids); + owner_relids = owner_in_outer ? outer_relids : inner_relids; + other_relids = owner_in_outer ? inner_relids : outer_relids; + + if (bms_is_subset(f->build_relids, owner_relids)) + { + /* + * Build side already sits with the owner; this shouldn't + * normally happen (such a filter would have been resolved at + * a lower join), but if it does, just propagate it unchanged. + * + * We still must be able to reach the owner's scan from above, + * so the owner has to be on a side this join preserves (see + * bloom_join_side_preserved); otherwise the filter could not + * be pushed to a recipient and this path must be rejected. + */ + if (!bloom_join_side_preserved(jointype, owner_in_outer)) + goto contradiction; + result = lappend(result, f); + } + else if (bms_is_subset(f->build_relids, join_relids)) + { + /* This join is the source of the filter. */ + if (is_hashjoin && + owner_in_outer && + bms_is_subset(f->build_relids, other_relids) && + jointype_realizes_bloom_filter(jointype) && + hashjoin_pushes_filter_to(hashclauses, outer_relids, + f->owner_relid)) + { + /* Realized by this hash join; drop from propagation. */ + if (realized != NULL) + *realized = lappend(*realized, f); + continue; + } + else + { + /* Cannot realize the filter here: reject this path. */ + goto contradiction; + } + } + else + { + /* + * Build side not yet available; propagate. As above, the + * filter can only reach its recipient scan if the owner stays + * on a side this join preserves; if not, reject this path so + * we never realize a filter with no recipient at plan time. + */ + if (!bloom_join_side_preserved(jointype, owner_in_outer)) + goto contradiction; + result = lappend(result, f); + } + } + } + + bms_free(join_relids); + return result; + +contradiction: + *contradicted = true; + if (realized != NULL) + { + list_free(*realized); + *realized = NIL; + } + bms_free(join_relids); + list_free(result); + return NIL; +} + +/* + * set_join_path_expected_filters + * Attach the propagated expected filters to a freshly created join path + * and reduce its row estimate to reflect their combined selectivity. + */ +static void +set_join_path_expected_filters(Path *path, List *filters) +{ + if (filters == NIL) + return; + + path->expected_filters = filters; + path->rows = clamp_row_est(path->rows * + expected_filters_selectivity(filters)); +} + /* * try_nestloop_path * Consider a nestloop join path; if it appears useful, push it into @@ -967,26 +1232,72 @@ try_nestloop_path(PlannerInfo *root, nestloop_subtype | PGS_CONSIDER_NONPARTIAL, outer_path, inner_path, extra); - if (add_path_precheck(joinrel, workspace.disabled_nodes, - workspace.startup_cost, workspace.total_cost, - pathkeys, required_outer)) - { - add_path(joinrel, (Path *) - create_nestloop_path(root, - joinrel, - jointype, - &workspace, - extra, - outer_path, - inner_path, - extra->restrictlist, - pathkeys, - required_outer)); - } - else + /* + * Account for expected Bloom filters carried by the input paths. A + * nestloop never builds a Bloom filter, so if it is the source of any + * expected filter the path is contradicted and must be rejected; + * otherwise the filters propagate to the resulting path. + */ { - /* Waste no memory when we reject a path here */ - bms_free(required_outer); + bool contradicted; + List *jfilters; + + jfilters = compute_join_expected_filters(root, outer_path, inner_path, + jointype, false, NIL, + &contradicted, NULL); + + /* + * Contradicted means the inner/outer paths expect this join to + * realize one of the expected filters, but a nestloop can't do that. + * So these input paths are incompatible with a nestloop. + */ + if (contradicted) + { + bms_free(required_outer); + return; + } + + /* + * If the path expects any filters, it's excluded from the cost + * pruning performed by add_path (so don't bother with + * add_path_precheck either). Once a path has all filters satisfied + * (or there were no filters), do the pruning as usual. + * + * XXX We don't want the "regular" paths without filters to get + * removed, because we need the option to pick from join algorithms. + * Paths with filters would likely win (simply because there are fewer + * rows), but they only work with hashjoins. However, maybe the + * hashjoin won't work for some reason (e.g. it wouldn't fit into + * work_mem). + * + * XXX Maybe it'd be cleaner to do this in add_path_precheck (i.e. + * make it return true for paths with expected filters). + */ + if (jfilters != NIL || + add_path_precheck(joinrel, workspace.disabled_nodes, + workspace.startup_cost, workspace.total_cost, + pathkeys, required_outer)) + { + Path *nlpath; + + nlpath = (Path *) create_nestloop_path(root, + joinrel, + jointype, + &workspace, + extra, + outer_path, + inner_path, + extra->restrictlist, + pathkeys, + required_outer); + set_join_path_expected_filters(nlpath, jfilters); + add_path(joinrel, nlpath); + } + else + { + /* Waste no memory when we reject a path here */ + bms_free(required_outer); + } } } @@ -1160,30 +1471,58 @@ try_mergejoin_path(PlannerInfo *root, outer_presorted_keys, extra); - if (add_path_precheck(joinrel, workspace.disabled_nodes, - workspace.startup_cost, workspace.total_cost, - pathkeys, required_outer)) - { - add_path(joinrel, (Path *) - create_mergejoin_path(root, - joinrel, - jointype, - &workspace, - extra, - outer_path, - inner_path, - extra->restrictlist, - pathkeys, - required_outer, - mergeclauses, - outersortkeys, - innersortkeys, - outer_presorted_keys)); - } - else + /* + * Account for expected Bloom filters carried by the input paths. A + * mergejoin never builds a Bloom filter, so it contradicts (and cannot + * use) any input path for which it would be the filter's source. + * Filter-bearing paths bypass the precheck, since their reduced cost + * isn't comparable to ordinary paths. + * + * XXX see the comments in try_nestloop_path + */ { - /* Waste no memory when we reject a path here */ - bms_free(required_outer); + bool contradicted; + List *jfilters; + + jfilters = compute_join_expected_filters(root, outer_path, inner_path, + jointype, false, NIL, + &contradicted, NULL); + if (contradicted) + { + bms_free(required_outer); + return; + } + + if (jfilters != NIL || + add_path_precheck(joinrel, workspace.disabled_nodes, + workspace.startup_cost, workspace.total_cost, + pathkeys, required_outer)) + { + Path *mjpath; + + mjpath = (Path *) create_mergejoin_path(root, + joinrel, + jointype, + &workspace, + extra, + outer_path, + inner_path, + extra->restrictlist, + pathkeys, + required_outer, + mergeclauses, + outersortkeys, + innersortkeys, + outer_presorted_keys); + set_join_path_expected_filters(mjpath, jfilters); + add_path(joinrel, mjpath); + } + else + { + /* Waste no memory when we reject a path here */ + bms_free(required_outer); + } + } } @@ -1314,27 +1653,63 @@ try_hashjoin_path(PlannerInfo *root, initial_cost_hashjoin(root, &workspace, jointype, hashclauses, outer_path, inner_path, extra, false); - if (add_path_precheck(joinrel, workspace.disabled_nodes, - workspace.startup_cost, workspace.total_cost, - NIL, required_outer)) - { - add_path(joinrel, (Path *) - create_hashjoin_path(root, - joinrel, - jointype, - &workspace, - extra, - outer_path, - inner_path, - false, /* parallel_hash */ - extra->restrictlist, - required_outer, - hashclauses)); - } - else + /* + * Account for expected Bloom filters carried by the input paths. A hash + * join builds and pushes down a Bloom filter, so it realizes (and removes + * from propagation) any expected filter for which it is the source; other + * filters propagate upward. Filter-bearing paths bypass the precheck. + */ { - /* Waste no memory when we reject a path here */ - bms_free(required_outer); + bool contradicted; + List *jfilters; + List *realized; + + jfilters = compute_join_expected_filters(root, outer_path, inner_path, + jointype, true, hashclauses, + &contradicted, &realized); + + /* XXX Can a hashjoin contradict a filter? Probably not. */ + if (contradicted) + { + bms_free(required_outer); + return; + } + + if (jfilters != NIL || + add_path_precheck(joinrel, workspace.disabled_nodes, + workspace.startup_cost, workspace.total_cost, + NIL, required_outer)) + { + Path *hjpath; + + hjpath = (Path *) create_hashjoin_path(root, + joinrel, + jointype, + &workspace, + extra, + outer_path, + inner_path, + false, /* parallel_hash */ + extra->restrictlist, + required_outer, + hashclauses); + set_join_path_expected_filters(hjpath, jfilters); + + /* + * Record the filters this hash join realizes, so + * create_hashjoin_plan can push exactly those down (and no + * others) at plan-creation time. + */ + ((HashPath *) hjpath)->realized_filters = realized; + + add_path(joinrel, hjpath); + } + else + { + /* Waste no memory when we reject a path here */ + bms_free(required_outer); + return; + } } } @@ -2316,6 +2691,33 @@ hash_inner_and_outer(PlannerInfo *root, } } + /* + * Also consider outer paths that carry expected Bloom filters. These + * are deliberately excluded from cheapest_startup/total_path and from + * cheapest_parameterized_paths (see set_cheapest), so we must iterate + * the full outer pathlist to find them. A hash join is able to build + * and push down the filters, so these paths are useful here even when + * they would be contradicted at a non-hash join. + */ + foreach(lc1, outerrel->pathlist) + { + Path *outerpath = (Path *) lfirst(lc1); + + if (outerpath->expected_filters == NIL) + continue; + + if (PATH_PARAM_BY_REL(outerpath, innerrel)) + continue; + + try_hashjoin_path(root, + joinrel, + outerpath, + cheapest_total_inner, + hashclauses, + jointype, + extra); + } + /* * If the joinrel is parallel-safe, we may be able to consider a * partial hash join. diff --git a/src/backend/optimizer/plan/createplan.c b/src/backend/optimizer/plan/createplan.c index 2c696ea0268..31cc9d7df4b 100644 --- a/src/backend/optimizer/plan/createplan.c +++ b/src/backend/optimizer/plan/createplan.c @@ -4709,6 +4709,230 @@ create_mergejoin_plan(PlannerInfo *root, return join_plan; } +/* + * BLOOM FILTER PUSHDOWN + * + * When a hash join is created as a path, we decide whether it should build a + * Bloom filter and push it down to a scan on its outer (probe) side. That + * decision - which filters are selective enough to be worth building, and + * which scan they can be pushed to - is made entirely while creating paths + * (see find_interesting_bloom_filters and compute_join_expected_filters in the + * optimizer); and the chosen filters are recorded on the HashPath as its + * realized_filters. Here we merely propagate that decision into the plan: we + * never reconsider whether a filter is worthwhile, and in particular we never + * push a filter that was not selected as interesting when creating paths. + * + * For each realized filter we locate the scan node for its owner relation in + * the outer subtree, append a BloomFilter (built from the hash keys belonging + * to that filter) to the scan's bloom_filters list, and increment the + * hashjoin's bloom_consumer_count. + * + * As setrefs hasn't run yet, the hash keys are still the raw Vars. So it's + * safe to compare var->varno against the scanrelid, and copy the keys verbatim + * onto the recipient. setrefs will rewrite the Vars later as usual, just like + * for the recipient's qual. + * + * XXX In most cases there'll be only a single consumer node. To get multiple + * consumers, we'd need either joins on the same keys, or ability to produce + * filters for subsets of the join keys (for cases where the join is more + * complex, and does not map to a single scan node directly). Seems like a + * possible future improvement. + * + * XXX Actually, we could have multiple consumer nodes for partitioned tables, + * where each partition gets a separate scan. Which seems like something we + * should support. + * + * XXX The recipient node must be one of a small set of scan nodes. We could + * relax this, and allow pushing to other nodes (e.g. joins or aggregates). + * Future improvement. + * + * XXX We don't currently push the same HashJoin to multiple recipients, but + * multiple HashJoins may attach a filter to the same scan node. + * -------------------------------------------------------------------------- + */ + +/* + * find_bloom_filter_recipient + * Try to find a scan node to push filter to. + * + * We support pushing filter to a subset of scan nodes (could be extended + * later). We support pushing filters through intermediate nodes (joins, + * sorts, ...). See bloom_join_side_preserved (joinpath.c) for joins; the + * path-time propagation uses the same predicate, so any filter realized while + * costing paths is guaranteed a reachable recipient here. + * + * XXX We could push filters through more nodes - e.g. aggregates if the + * hash keys match GROUP BY keys (are a subset of). + * + * XXX We could do pushdown to parallel parts of a query. But we'd need + * a different way to communicate if a filter is built etc. (the worker + * won't have access to the hashjoin state). + * + * XXX Not sure this handles partitioned tables correctly. Those will be below + * Append node, and we don't push through those. But the scans will still expect + * the filter, I think. Even if we pushed through Append node, it probably won't + * work because we expect a single consumer. But we'll have one consumer per + * scan of a partition. + */ +static Plan * +find_bloom_filter_recipient(Plan *plan, Index target_relid) +{ + /* XXX shouldn't really happen, I think */ + if (plan == NULL) + return NULL; + + /* XXX no pushdown to parallel parts of a query */ + if (plan->parallel_aware) + return NULL; + + switch (nodeTag(plan)) + { + case T_SeqScan: + case T_SampleScan: + case T_IndexScan: + case T_IndexOnlyScan: + case T_BitmapHeapScan: + case T_TidScan: + case T_TidRangeScan: + { + Scan *scan = (Scan *) plan; + + if (scan->scanrelid == target_relid) + return plan; + return NULL; + } + case T_Sort: + case T_IncrementalSort: + case T_Material: + case T_Memoize: + case T_Unique: + case T_Limit: + return find_bloom_filter_recipient(outerPlan(plan), target_relid); + case T_HashJoin: + case T_NestLoop: + case T_MergeJoin: + { + JoinType jt = ((Join *) plan)->jointype; + Plan *res; + + /* + * We can only push to one node, but we don't know on which + * side of the join it is (e.g. for inner joins it can be on + * both sides). So make sure to check both, if compatible with + * the join type (per bloom_join_side_preserved). + */ + if (bloom_join_side_preserved(jt, true)) + { + res = find_bloom_filter_recipient(outerPlan(plan), + target_relid); + if (res != NULL) + return res; + } + if (bloom_join_side_preserved(jt, false)) + { + res = find_bloom_filter_recipient(innerPlan(plan), + target_relid); + if (res != NULL) + return res; + } + return NULL; + } + default: + return NULL; + } +} + +/* + * try_push_bloom_filter + * Push down the bloom filter the planner decided this hashjoin should + * build, recording it on the recipient scan node. + * + * 'realized_filters' is the list of ExpectedFilter nodes the HashPath was + * found to realize while creating paths. We do not reconsider that decision + * here; if it is empty, this hashjoin pushes nothing. Otherwise we turn it + * into a concrete BloomFilter attached to the scan of the owner relation. + * + * A hash join builds a single bloom filter, populated with the combined hash + * value of all of its hash keys (see ExecHashTableInsert / bloom_add_element + * in nodeHash.c). The recipient must therefore probe with the matching full + * set of outer hash keys, so we build the filter from all of them. All + * realized filters of one hash join share the same owner relation (the outer + * side of every hash clause is a bare Var of that relation), so a single + * pushed-down filter covers them all. + * + * Note the pushdown still happens during plan creation, i.e. after the plan was + * already selected. The selectivity of the filter was, however, accounted for + * while creating paths (the affected scan paths carry reduced row estimates), + * so the plan-time row counts already reflect the expected elimination. + */ +static void +try_push_bloom_filter(PlannerInfo *root, HashJoin *hj, Plan *outer_plan, + List *realized_filters) +{ + ExpectedFilter *f; + Index owner_relid; + Plan *recipient; + BloomFilter *bf; + + /* Nothing to do unless the planner chose to realize a filter here. */ + if (realized_filters == NIL) + return; + + /* + * The feature must have been enabled when paths were built; otherwise no + * filter would have been realized. + */ + Assert(enable_hashjoin_bloom); + Assert(hj->hashkeys != NIL); + Assert(list_length(hj->hashkeys) == list_length(hj->hashoperators)); + Assert(list_length(hj->hashkeys) == list_length(hj->hashcollations)); + + /* + * All realized filters share the same owner relation (every hash clause's + * outer side is a bare Var of that relation). + */ + f = (ExpectedFilter *) linitial(realized_filters); + owner_relid = f->owner_relid; + +#ifdef USE_ASSERT_CHECKING + { + ListCell *lc; + + foreach(lc, realized_filters) + Assert(((ExpectedFilter *) lfirst(lc))->owner_relid == owner_relid); + } +#endif + + /* + * Locate the scan node for the owner relation in the outer subtree. The + * path machinery guaranteed such a recipient exists (the filter could not + * have been realized otherwise), but stay defensive. + */ + recipient = find_bloom_filter_recipient(outer_plan, owner_relid); + if (recipient == NULL) + return; + + /* + * Assign the filter an ID. We'll use it to register the filter in + * ExecRegisterBloomFilterProducer, and then for lookups in + * LookupBloomFilterProducer during execution. + * + * XXX We can't use plan_node_id, as it's not assigned yet, that only + * happens in set_plan_refs. + */ + hj->bloom_filter_id = ++root->glob->lastBloomFilterId; + + bf = makeNode(BloomFilter); + bf->filter_exprs = (List *) copyObject(hj->hashkeys); + bf->hashops = list_copy(hj->hashoperators); + bf->hashcollations = list_copy(hj->hashcollations); + bf->producer_id = hj->bloom_filter_id; + + recipient->bloom_filters = lappend(recipient->bloom_filters, bf); + + hj->bloom_consumer_count++; +} + static HashJoin * create_hashjoin_plan(PlannerInfo *root, HashPath *best_path) @@ -4886,6 +5110,16 @@ create_hashjoin_plan(PlannerInfo *root, copy_generic_path_info(&join_plan->join.plan, &best_path->jpath.path); + /* + * Propagate the bloom filter pushdown decision made while creating paths: + * if this hash join was found to realize one or more bloom filters, push + * the corresponding filter down to the recipient scan in the outer + * subtree, and bump our bloom_consumer_count. No filter that was not + * selected as interesting during path creation is pushed here. + */ + try_push_bloom_filter(root, join_plan, outer_plan, + best_path->realized_filters); + return join_plan; } diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 3225185d16f..8bb0948c4b8 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -384,6 +384,7 @@ standard_planner(Query *parse, const char *query_string, int cursorOptions, glob->lastPHId = 0; glob->lastRowMarkId = 0; glob->lastPlanNodeId = 0; + glob->lastBloomFilterId = 0; glob->transientPlan = false; glob->dependsOnRole = false; glob->partition_directory = NULL; diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c index e1182255055..76f7560cceb 100644 --- a/src/backend/optimizer/plan/setrefs.c +++ b/src/backend/optimizer/plan/setrefs.c @@ -159,6 +159,7 @@ static Node *fix_scan_expr(PlannerInfo *root, Node *node, int rtoffset, double num_exec); static Node *fix_scan_expr_mutator(Node *node, fix_scan_expr_context *context); static bool fix_scan_expr_walker(Node *node, fix_scan_expr_context *context); +static void fix_scan_bloom_filters(PlannerInfo *root, Plan *plan, int rtoffset); static void set_join_references(PlannerInfo *root, Join *join, int rtoffset); static void set_upper_references(PlannerInfo *root, Plan *plan, int rtoffset); static void set_param_references(PlannerInfo *root, Plan *plan); @@ -662,6 +663,9 @@ set_plan_refs(PlannerInfo *root, Plan *plan, int rtoffset) splan->scan.plan.qual = fix_scan_list(root, splan->scan.plan.qual, rtoffset, NUM_EXEC_QUAL(plan)); + + /* pushed-down bloom filters */ + fix_scan_bloom_filters(root, plan, rtoffset); } break; case T_SampleScan: @@ -678,6 +682,9 @@ set_plan_refs(PlannerInfo *root, Plan *plan, int rtoffset) splan->tablesample = (TableSampleClause *) fix_scan_expr(root, (Node *) splan->tablesample, rtoffset, 1); + + /* pushed-down bloom filters */ + fix_scan_bloom_filters(root, plan, rtoffset); } break; case T_IndexScan: @@ -703,6 +710,9 @@ set_plan_refs(PlannerInfo *root, Plan *plan, int rtoffset) splan->indexorderbyorig = fix_scan_list(root, splan->indexorderbyorig, rtoffset, NUM_EXEC_QUAL(plan)); + + /* pushed-down bloom filters */ + fix_scan_bloom_filters(root, plan, rtoffset); } break; case T_IndexOnlyScan: @@ -741,6 +751,9 @@ set_plan_refs(PlannerInfo *root, Plan *plan, int rtoffset) splan->bitmapqualorig = fix_scan_list(root, splan->bitmapqualorig, rtoffset, NUM_EXEC_QUAL(plan)); + + /* pushed-down bloom filters */ + fix_scan_bloom_filters(root, plan, rtoffset); } break; case T_TidScan: @@ -757,6 +770,9 @@ set_plan_refs(PlannerInfo *root, Plan *plan, int rtoffset) splan->tidquals = fix_scan_list(root, splan->tidquals, rtoffset, 1); + + /* pushed-down bloom filters */ + fix_scan_bloom_filters(root, plan, rtoffset); } break; case T_TidRangeScan: @@ -773,6 +789,9 @@ set_plan_refs(PlannerInfo *root, Plan *plan, int rtoffset) splan->tidrangequals = fix_scan_list(root, splan->tidrangequals, rtoffset, 1); + + /* pushed-down bloom filters */ + fix_scan_bloom_filters(root, plan, rtoffset); } break; case T_SubqueryScan: @@ -1420,6 +1439,30 @@ set_indexonlyscan_references(PlannerInfo *root, INDEX_VAR, rtoffset, NUM_EXEC_QUAL((Plan *) plan)); + + /* + * Bloom filter pushdown: any BloomFilter recipient lists also need their + * key expressions rewritten to reference the index tuple, since that's + * what the recipient scan returns and what ExecBloomFilters will evaluate + * against. + */ + if (plan->scan.plan.bloom_filters != NIL) + { + ListCell *lc2; + + foreach(lc2, plan->scan.plan.bloom_filters) + { + BloomFilter *bf = lfirst_node(BloomFilter, lc2); + + bf->filter_exprs = (List *) + fix_upper_expr(root, + (Node *) bf->filter_exprs, + index_itlist, + INDEX_VAR, + rtoffset, + NUM_EXEC_QUAL((Plan *) plan)); + } + } /* indexqual is already transformed to reference index columns */ plan->indexqual = fix_scan_list(root, plan->indexqual, rtoffset, 1); @@ -2378,6 +2421,26 @@ fix_scan_expr_walker(Node *node, fix_scan_expr_context *context) return expression_tree_walker(node, fix_scan_expr_walker, context); } +/* + * Fix references in hashkey expressions of bloom filters pushed down. + */ +static void +fix_scan_bloom_filters(PlannerInfo *root, Plan *plan, int rtoffset) +{ + ListCell *lc; + + if (plan->bloom_filters == NIL) + return; + + foreach(lc, plan->bloom_filters) + { + BloomFilter *bf = lfirst_node(BloomFilter, lc); + + bf->filter_exprs = fix_scan_list(root, bf->filter_exprs, rtoffset, + NUM_EXEC_QUAL(plan)); + } +} + /* * set_join_references * Modify the target list and quals of a join node to reference its diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c index 3875e3dd571..e1c36fcb533 100644 --- a/src/backend/optimizer/util/pathnode.c +++ b/src/backend/optimizer/util/pathnode.c @@ -286,6 +286,16 @@ set_cheapest(RelOptInfo *parent_rel) Path *path = (Path *) lfirst(p); int cmp; + /* + * Paths that expect a pushed-down Bloom filter are speculative: their + * rows/cost estimates assume a hash join above will build and push a + * filter to them. They must never be chosen as the cheapest startup, + * total, or parameterized path; they are only consumed explicitly by + * join path generation (see joinpath.c). Skip them here. + */ + if (path->expected_filters != NIL) + continue; + if (path->param_info) { /* Parameterized path, so add it to parameterized_paths */ @@ -383,6 +393,129 @@ set_cheapest(RelOptInfo *parent_rel) parent_rel->cheapest_parameterized_paths = parameterized_paths; } +/* + * expected_filters_equal + * Return true if the two lists of ExpectedFilter nodes denote the same + * set of expected Bloom filters (order-independent). + */ +bool +expected_filters_equal(List *a, List *b) +{ + ListCell *lc; + + if (a == NIL && b == NIL) + return true; + if (list_length(a) != list_length(b)) + return false; + + foreach(lc, a) + { + ExpectedFilter *fa = (ExpectedFilter *) lfirst(lc); + ListCell *lc2; + bool found = false; + + foreach(lc2, b) + { + ExpectedFilter *fb = (ExpectedFilter *) lfirst(lc2); + + if (fa->owner_relid == fb->owner_relid && + bms_equal(fa->build_relids, fb->build_relids) && + equal(fa->clauses, fb->clauses)) + { + found = true; + break; + } + } + if (!found) + return false; + } + return true; +} + +/* + * expected_filters_selectivity + * Combined surviving fraction of a set of expected filters, assuming + * independence. Returns a value in (0, 1]. + */ +double +expected_filters_selectivity(List *filters) +{ + double sel = 1.0; + ListCell *lc; + + foreach(lc, filters) + { + ExpectedFilter *f = (ExpectedFilter *) lfirst(lc); + + sel *= f->selectivity; + } + + /* clamp to a sane range */ + if (sel < 0.0) + sel = 0.0; + if (sel > 1.0) + sel = 1.0; + + return sel; +} + +/* + * create_filtered_scan_path + * Build a copy of a base-relation scan path that additionally expects the + * given set of pushed-down Bloom filters. + * + * The clone shares all substructure with the original path (parent, + * pathtarget, clauses, etc.); only the rows estimate is reduced to reflect + * the filters' combined selectivity, and expected_filters is set. This is + * safe because create_plan() treats the clone identically to the original + * (it ignores expected_filters), and add_path() may freely pfree the clone. + * + * Only the plain scan path node types that can receive a pushed-down filter + * are supported (matching find_bloom_filter_recipient in createplan.c). + * Returns NULL for unsupported path types. + * + * XXX This should probably adjust the CPU cost in some way. It assumes the + * filter checks are free, which does not seem right. + */ +Path * +create_filtered_scan_path(PlannerInfo *root, Path *subpath, List *filters) +{ + Path *newpath; + size_t sz; + + switch (nodeTag(subpath)) + { + case T_Path: + /* plain seqscan/samplescan etc. */ + sz = sizeof(Path); + break; + case T_IndexPath: + sz = sizeof(IndexPath); + break; + case T_BitmapHeapPath: + sz = sizeof(BitmapHeapPath); + break; + case T_TidPath: + sz = sizeof(TidPath); + break; + case T_TidRangePath: + sz = sizeof(TidRangePath); + break; + default: + /* unsupported scan path type */ + return NULL; + } + + newpath = (Path *) palloc(sz); + memcpy(newpath, subpath, sz); + + newpath->expected_filters = filters; + newpath->rows = clamp_row_est(subpath->rows * + expected_filters_selectivity(filters)); + + return newpath; +} + /* * add_path * Consider a potential implementation path for the specified parent rel, @@ -485,6 +618,17 @@ add_path(RelOptInfo *parent_rel, Path *new_path) PathKeysComparison keyscmp; BMS_Comparison outercmp; + /* + * Paths carrying different sets of expected Bloom filters serve + * different purposes (each may be consumed by a different parent + * join, or none at all), and their cost/row estimates aren't directly + * comparable. So if the two paths don't expect the same filters, + * keep both and don't let either dominate the other. + */ + if (!expected_filters_equal(new_path->expected_filters, + old_path->expected_filters)) + continue; + /* * Do a fuzzy cost comparison with standard fuzziness limit. */ @@ -702,6 +846,20 @@ add_path_precheck(RelOptInfo *parent_rel, int disabled_nodes, Path *old_path = (Path *) lfirst(p1); PathKeysComparison keyscmp; + /* + * Paths carrying expected Bloom filters serve a different purpose and + * are not directly cost-comparable with ordinary paths, exactly as in + * add_path (which keeps both when the expected filter sets differ). + * The candidates submitted to this precheck never carry expected + * filters of their own, so any filter-bearing old path is a + * non-comparable speculative path and must not be allowed to dominate + * (and thereby suppress) the new path. Skipping them here also + * guarantees that a join relation always retains at least one + * ordinary, filter-free path to serve as cheapest_total_path. + */ + if (old_path->expected_filters != NIL) + continue; + /* * Since the pathlist is sorted by disabled_nodes and then by * total_cost, we can stop looking once we reach a path with more diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat index d421cdbde76..aacadc8de40 100644 --- a/src/backend/utils/misc/guc_parameters.dat +++ b/src/backend/utils/misc/guc_parameters.dat @@ -380,6 +380,26 @@ max => 'BLCKSZ', }, +{ name => 'bloom_filter_pushdown_max', type => 'int', context => 'PGC_USERSET', group => 'QUERY_TUNING_OTHER', + short_desc => 'Maximum number of pushed-down hash join bloom filters considered per scan.', + long_desc => 'Bounds how many interesting bloom filters the planner enumerates subsets of when building filter-aware scan paths.', + flags => 'GUC_EXPLAIN', + variable => 'bloom_filter_pushdown_max', + boot_val => '3', + min => '0', + max => '10', +}, + +{ name => 'bloom_filter_pushdown_threshold', type => 'real', context => 'PGC_USERSET', group => 'QUERY_TUNING_OTHER', + short_desc => 'Minimum fraction of tuples a pushed-down hash join bloom filter must be expected to eliminate.', + long_desc => 'A bloom filter is only considered during planning if it is expected to discard at least this fraction of the scanned tuples.', + flags => 'GUC_EXPLAIN', + variable => 'bloom_filter_pushdown_threshold', + boot_val => '0.3', + min => '0.0', + max => '1.0', +}, + { name => 'bonjour', type => 'bool', context => 'PGC_POSTMASTER', group => 'CONN_AUTH_SETTINGS', short_desc => 'Enables advertising the server via Bonjour.', variable => 'enable_bonjour', @@ -926,6 +946,13 @@ boot_val => 'true', }, +{ name => 'enable_hashjoin_bloom', type => 'bool', context => 'PGC_USERSET', group => 'QUERY_TUNING_METHOD', + short_desc => 'Enables hash join bloom filter pushdown to the outer side.', + flags => 'GUC_EXPLAIN', + variable => 'enable_hashjoin_bloom', + boot_val => 'true', +}, + { name => 'enable_incremental_sort', type => 'bool', context => 'PGC_USERSET', group => 'QUERY_TUNING_METHOD', short_desc => 'Enables the planner\'s use of incremental sort steps.', flags => 'GUC_EXPLAIN', diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index 7958653077b..3247ed46e6e 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -449,6 +449,7 @@ #enable_distinct_reordering = on #enable_self_join_elimination = on #enable_eager_aggregate = on +#enable_hashjoin_bloom = on # - Planner Cost Constants - @@ -485,6 +486,8 @@ # - Other Planner Options - +#bloom_filter_pushdown_max = 3 # range 0-10 +#bloom_filter_pushdown_threshold = 0.3 # range 0.0-1.0 #default_statistics_target = 100 # range 1-10000 #constraint_exclusion = partition # on, off, or partition #cursor_tuple_fraction = 0.1 # range 0.0-1.0 diff --git a/src/include/executor/execScan.h b/src/include/executor/execScan.h index 25efc622f88..8c2ec0de292 100644 --- a/src/include/executor/execScan.h +++ b/src/include/executor/execScan.h @@ -170,10 +170,11 @@ ExecScanExtended(ScanState *node, /* interrupt checks are in ExecScanFetch */ /* - * If we have neither a qual to check nor a projection to do, just skip - * all the overhead and return the raw scan tuple. + * If we have neither a qual to check nor a projection to do nor any + * pushed-down bloom filter to probe, just skip all the overhead and + * return the raw scan tuple. */ - if (!qual && !projInfo) + if (!qual && !projInfo && node->ps.bloom_filters == NIL) { ResetExprContext(econtext); return ExecScanFetch(node, epqstate, accessMtd, recheckMtd); @@ -214,6 +215,21 @@ ExecScanExtended(ScanState *node, */ econtext->ecxt_scantuple = slot; + /* + * If runtime bloom filters have been pushed down to this scan, check + * them first. A rejected tuple is dropped silently (no "Rows Removed + * by Filter" instrumentation -- the per-filter counters in + * BloomFilterState already capture this). + * + * XXX Maybe we should include this in "Rows Removed by Filter"? + */ + if (node->ps.bloom_filters != NIL && + !ExecBloomFilters(node->ps.bloom_filters, econtext)) + { + ResetExprContext(econtext); + continue; + } + /* * check that the current tuple satisfies the qual-clause * diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 1798e6027d4..12b0b2faedf 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -514,6 +514,18 @@ ExecProject(ProjectionInfo *projInfo) } #endif +/* + * Bloom filter pushdown (see nodeHashjoin.c). Declared here so that + * scan nodes that act as recipients don't need to pull in hashjoin + * internals just to call these helpers from their ExecInit. + * + * XXX There's probably a better place for this. It should live in + * the executor somewhere, not in nodeHashjoin.c? + */ +extern bool ExecBloomFilters(List *filters, ExprContext *econtext); +extern void ExecInitBloomFilters(PlanState *planstate, + TupleTableSlot *output_slot); + /* * ExecQual - evaluate a qual prepared with ExecInitQual (possibly via * ExecPrepareQual). Returns true if qual is satisfied, else false. diff --git a/src/include/executor/nodeHashjoin.h b/src/include/executor/nodeHashjoin.h index aebd39be8b5..08efefae209 100644 --- a/src/include/executor/nodeHashjoin.h +++ b/src/include/executor/nodeHashjoin.h @@ -31,4 +31,13 @@ extern void ExecHashJoinInitializeWorker(HashJoinState *state, extern void ExecHashJoinSaveTuple(MinimalTuple tuple, uint32 hashvalue, BufFile **fileptr, HashJoinTable hashtable); +/* + * Bloom filter pushdown producer-side helper (see nodeHashjoin.c). + * + * ExecBloomFilters and ExecInitBloomFilters live in executor.h so that + * scan nodes can call them from ExecInit without having to pull in + * hashjoin internals. + */ +extern void ExecRegisterBloomFilterProducer(HashJoinState *hjstate); + #endif /* NODEHASHJOIN_H */ diff --git a/src/include/lib/bloomfilter.h b/src/include/lib/bloomfilter.h index 860ee9bdc72..9277087acb6 100644 --- a/src/include/lib/bloomfilter.h +++ b/src/include/lib/bloomfilter.h @@ -23,5 +23,7 @@ extern void bloom_add_element(bloom_filter *filter, unsigned char *elem, extern bool bloom_lacks_element(bloom_filter *filter, unsigned char *elem, size_t len); extern double bloom_prop_bits_set(bloom_filter *filter); +extern uint64 bloom_total_bits(bloom_filter *filter); +extern int bloom_hash_funcs(bloom_filter *filter); #endif /* BLOOMFILTER_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e64fd8c7ea3..5e08b356d9f 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -763,6 +763,14 @@ typedef struct EState List *es_auxmodifytables; /* List of secondary ModifyTableStates */ + /* + * List of nodes producing pushed-down bloom filters. Each list element is + * a HashJoinState with at least one filter pushed down to a scan node. + * Scans look up their producer in this list (by plan_node_id) lazily on + * first probe (and cache the result). + */ + List *es_bloom_producers; + /* * this ExprContext is for per-output-tuple operations, such as constraint * checks and index-value computations. It will be reset for each output @@ -1287,8 +1295,58 @@ typedef struct PlanState bool outeropsset; bool inneropsset; bool resultopsset; + + /* + * Bloom filters (BloomFilterState) pushed to this node (NIL if no filters + * were pushed down to this node). Right now only some scan nodes expect + * filters, but the list is in PlanState so that we can expand the feature + * to more nodes in the future. + */ + List *bloom_filters; } PlanState; +/* + * BloomFilterState + * + * State for executing (evaluating) a BloomFilter, pushed to a scan node. + * + * The producer is the HashJoinState this bloom filter is for. It's resolved + * at executor-init time using EState.es_bloom_producers (the producer always + * runs ExecInit before its recipients). + * + * 'checked' and 'rejected' are per-recipient counters reported by EXPLAIN + * ANALYZE. It's a bit redundant with the fields we have in the producer + * (see HashState). But if we ever end up with multiple consumers per filter, + * it'd be useful. + * + * The adaptive fields track the match fraction in recently checked probes. + * When most checks are passing through, the executor starts probing less + * frequently (and sampling probes instead), until the fraction of matches + * drops below some threshold. + */ +typedef struct BloomFilterState +{ + NodeTag type; + + /* producer HJ node and the filter */ + int producer_id; + struct HashJoinState *producer; + BloomFilter *filter; + + int nkeys; /* number of hash keys */ + ExprState *keys; /* ExprState for the hash calculation */ + + /* probe counters */ + uint64 checked; + uint64 rejected; + + /* adaptive probing */ + uint64 adaptiveWindowProbes; + uint64 adaptiveWindowMatches; + uint64 adaptiveSampleCounter; + bool adaptiveSampling; +} BloomFilterState; + /* ---------------- * these are defined to avoid confusion problems with "left" * and "right" and "inner" and "outer". The convention is that @@ -2703,6 +2761,39 @@ typedef struct HashState Tuplestorestate *null_tuple_store; /* where to put null-keyed tuples */ bool keep_null_tuples; /* do we need to save such tuples? */ + /* + * True iff at we managed to push down the bloom filter to at least one + * node on the HashJoin's outer side. It tells the Hash node to also build + * a bloom filter next to the hash table. + */ + bool want_bloom_filter; + + /* + * Mirror of the parent HashJoin's bloom_filter_id, copied here at + * ExecInitHashJoin time so EXPLAIN's show_hash_info can label the filter + * without traversing back up the plan-state tree (which is not easy, as + * there's no parent in PlanState). + */ + int bloom_filter_id; + + /* + * Bloom filter on hash values during the build phase. Probed by recipient + * nodes (typically scans in the outer subtree). NULL when pushdown is + * disabled or no recipient was identified, and (transiently) before + * ExecHashTableCreate runs (on the first outer tuple). + * + * Allocated in hashCxt, just like the hashtable itself. Reset on rescans. + */ + struct bloom_filter *bloom_filter; + + /* + * Counters with total per-filter instrumentation. Separate from the + * per-recipient counters in BloomFilterState. Redundant, but will be + * needed if we end up allowing multiple recipients. + */ + uint64 bloomFilterChecked; + uint64 bloomFilterRejected; + /* * In a parallelized hash join, the leader retains a pointer to the * shared-memory stats area in its shared_info field, and then copies the diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 27a2c6815b7..2dbbc241420 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -244,6 +244,9 @@ typedef struct PlannerGlobal /* highest plan node ID assigned */ int lastPlanNodeId; + /* highest bloom-filter-producer ID assigned (see plannodes.h) */ + int lastBloomFilterId; + /* redo plan when TransactionXmin changes? */ bool transientPlan; @@ -1833,6 +1836,54 @@ typedef struct GroupByOrdering List *clauses; } GroupByOrdering; +/* + * ExpectedFilter + * + * Represents the planner's assumption that a scan path will receive a Bloom + * filter pushed down at plan-creation time by some hash join above it (see + * the bloom filter pushdown logic in createplan.c and nodeHashjoin.c). When + * a scan participates in a hashjoinable equality join, the hash join may build + * a Bloom filter on the inner ("build") side keys and push it to the scan on + * the outer ("probe") side, eliminating outer tuples that cannot match. + * + * Because that pushdown happens after path selection, the row/cost estimates + * of the affected paths would normally ignore the filter's selectivity. To + * account for it during planning, we generate additional scan paths that carry + * one or more ExpectedFilter nodes and whose rows/cost reflect the expected + * elimination. These filters are propagated up the join tree much like + * pathkeys, until the hash join that is their "source" realizes them. + * + * 'owner_relid' is the base relation the filter would be pushed down to. + * 'build_relids' is the set of base relids on the other (build) side of the + * join clause(s); the filter is "sourced" by the join that first brings + * those relids together with the owner. + * 'clauses' is the list of hashjoinable equality RestrictInfos defining the + * filter keys (the owner side of each clause is a bare Var of owner_relid). + * 'selectivity' is the expected surviving fraction of owner rows (in (0,1]). + * + * XXX The "owner_relid" may be a bit misleading, particularly if we allow + * pushing a filter to multiple nodes (e.g. scans on a partition). In that case + * we'd have multiple "owners", but ownership suggests there's just one. And + * some places already use "consumer" when referencing to the scan nodes, so + * maybe we should just use that? + * + * XXX What if we allow pushdown to non-scan nodes, e.g. above a join when + * pushdown to a scan is not possible (e.g. because the join clause is complex + * and references multiple relations)? Or maybe we could push to ForeignScan, + * and I'm not sure if those have a valid relid in all cases. + */ +typedef struct ExpectedFilter +{ + pg_node_attr(no_read, no_query_jumble) + + NodeTag type; + + Index owner_relid; + Relids build_relids; + List *clauses pg_node_attr(copy_as_scalar, equal_as_scalar); + Selectivity selectivity; +} ExpectedFilter; + /* * VolatileFunctionStatus -- allows nodes to cache their * contain_volatile_functions properties. VOLATILITY_UNKNOWN means not yet @@ -2009,6 +2060,14 @@ typedef struct Path /* sort ordering of path's output; a List of PathKey nodes; see above */ List *pathkeys; + + /* + * Bloom filters this path expects to receive from some hash join above it + * (a List of ExpectedFilter nodes; see below). An empty list means the + * path makes no such assumption. The path's rows/cost estimates already + * reflect the expected selectivity of these filters. + */ + List *expected_filters; } Path; /* Macro for extracting a path's parameterization relids; beware double eval */ @@ -2490,6 +2549,16 @@ typedef struct HashPath List *path_hashclauses; /* join clauses used for hashing */ int num_batches; /* number of batches expected */ Cardinality inner_rows_total; /* total inner rows expected */ + + /* + * Bloom filters this hash join "realizes": expected filters (see + * ExpectedFilter) carried by an input path for which this hash join is + * the source, and which it can push down to the scan of the owner + * relation. The pushdown decision is made here, while building paths; + * create_plan() merely propagates it to the plan (see + * create_hashjoin_plan). NIL means this hash join pushes no filter. + */ + List *realized_filters; } HashPath; /* diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h index b880ce0f4be..81e053473bb 100644 --- a/src/include/nodes/plannodes.h +++ b/src/include/nodes/plannodes.h @@ -254,8 +254,40 @@ typedef struct Plan */ Bitmapset *extParam; Bitmapset *allParam; + + /* + * List of BloomFilter nodes (or NIL). When non-empty, this plan node is + * a recipient of one or more runtime bloom filters pushed down at plan + * time by some HashJoin ancestor; see nodeHashjoin.c. Living on Plan + * (rather than on a specific subtype like Scan) allows pushdown to any + * node type that's prepared to call ExecBloomFilters on its output. + */ + List *bloom_filters; } Plan; +/* + * BloomFilter - + * One pushed-down bloom filter, attached to a recipient Plan node. + * + * 'filter_exprs', 'hashops' and 'hashcollations' are parallel lists, one + * entry per join key: the expression to hash, its hash operator (OID), + * and its input collation (OID). + * + * 'producer_id' is the bloom_filter_id of the producing HashJoin (resolved at + * executor init time via EState.es_bloom_producers). + * ---------------- + */ +typedef struct BloomFilter +{ + pg_node_attr(no_query_jumble) + + NodeTag type; + List *filter_exprs; + List *hashops; + List *hashcollations; + int producer_id; +} BloomFilter; + /* ---------------- * these are defined to avoid confusion problems with "left" * and "right" and "inner" and "outer". The convention is that @@ -1075,6 +1107,26 @@ typedef struct HashJoin * perform lookups in the hashtable over the inner plan. */ List *hashkeys; + + /* + * Number of plan nodes that consume this HashJoin's bloom filter via + * pushdown. Set at plan time by the bloom filter pushdown pass. + * + * At execution time, the HashJoin builds the bloom filter only when this + * is non-zero (and the enable_hashjoin_bloom GUC is on). + */ + int bloom_consumer_count; + + /* + * Identifier used by recipient nodes to find this producer at execution + * time, via EState.es_bloom_producers. Assigned during + * create_hashjoin_plan from PlannerGlobal.lastBloomFilterId. Each + * BloomFilter on a recipient stores a copy in its producer_id field for + * convenience. + * + * Zero when this HashJoin has no consumers. + */ + int bloom_filter_id; } HashJoin; /* ---------------- diff --git a/src/include/optimizer/cost.h b/src/include/optimizer/cost.h index bda3f1690c0..2047f211c4c 100644 --- a/src/include/optimizer/cost.h +++ b/src/include/optimizer/cost.h @@ -63,6 +63,7 @@ extern PGDLLIMPORT bool enable_material; extern PGDLLIMPORT bool enable_memoize; extern PGDLLIMPORT bool enable_mergejoin; extern PGDLLIMPORT bool enable_hashjoin; +extern PGDLLIMPORT bool enable_hashjoin_bloom; extern PGDLLIMPORT bool enable_gathermerge; extern PGDLLIMPORT bool enable_partitionwise_join; extern PGDLLIMPORT bool enable_partitionwise_aggregate; @@ -71,6 +72,8 @@ extern PGDLLIMPORT bool enable_parallel_hash; extern PGDLLIMPORT bool enable_partition_pruning; extern PGDLLIMPORT bool enable_presorted_aggregate; extern PGDLLIMPORT bool enable_async_append; +extern PGDLLIMPORT double bloom_filter_pushdown_threshold; +extern PGDLLIMPORT int bloom_filter_pushdown_max; extern PGDLLIMPORT int constraint_exclusion; extern double index_pages_fetched(double tuples_fetched, BlockNumber pages, diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h index e8db321f92b..0dcae2ae3b3 100644 --- a/src/include/optimizer/pathnode.h +++ b/src/include/optimizer/pathnode.h @@ -64,6 +64,11 @@ extern bool add_partial_path_precheck(RelOptInfo *parent_rel, int disabled_nodes, Cost startup_cost, Cost total_cost, List *pathkeys); +extern bool expected_filters_equal(List *a, List *b); +extern double expected_filters_selectivity(List *filters); +extern Path *create_filtered_scan_path(PlannerInfo *root, Path *subpath, + List *filters); + extern Path *create_seqscan_path(PlannerInfo *root, RelOptInfo *rel, Relids required_outer, int parallel_workers); extern Path *create_samplescan_path(PlannerInfo *root, RelOptInfo *rel, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 17f2099ec3b..636761f2e94 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -104,6 +104,7 @@ extern void add_paths_to_joinrel(PlannerInfo *root, RelOptInfo *joinrel, RelOptInfo *outerrel, RelOptInfo *innerrel, JoinType jointype, SpecialJoinInfo *sjinfo, List *restrictlist); +extern bool bloom_join_side_preserved(JoinType jointype, bool to_outer); /* * joinrels.c @@ -198,6 +199,11 @@ extern List *generate_implied_equalities_for_column(PlannerInfo *root, ec_matches_callback_type callback, void *callback_arg, Relids prohibited_rels); +extern List *generate_implied_equalities_for_all_columns(PlannerInfo *root, + RelOptInfo *rel, + ec_matches_callback_type callback, + void *callback_arg, + Relids prohibited_rels); extern bool have_relevant_eclass_joinclause(PlannerInfo *root, RelOptInfo *rel1, RelOptInfo *rel2); extern bool has_relevant_eclass_joinclause(PlannerInfo *root, diff --git a/src/test/regress/expected/aggregates.out b/src/test/regress/expected/aggregates.out index 5f0668382ed..f7cf71f3993 100644 --- a/src/test/regress/expected/aggregates.out +++ b/src/test/regress/expected/aggregates.out @@ -1539,9 +1539,11 @@ group by t1.a,t1.b,t1.c,t1.d,t2.x,t2.y,t2.z; -> Hash Join Hash Cond: ((t2.x = t1.a) AND (t2.y = t1.b)) -> Seq Scan on t2 + Bloom Filter 1: keys=(x, y) -> Hash + Bloom Filter 1 -> Seq Scan on t1 -(7 rows) +(9 rows) -- Test case where t1 can be optimized but not t2 explain (costs off) select t1.*,t2.x,t2.z @@ -1554,9 +1556,11 @@ group by t1.a,t1.b,t1.c,t1.d,t2.x,t2.z; -> Hash Join Hash Cond: ((t2.x = t1.a) AND (t2.y = t1.b)) -> Seq Scan on t2 + Bloom Filter 1: keys=(x, y) -> Hash + Bloom Filter 1 -> Seq Scan on t1 -(7 rows) +(9 rows) -- Cannot optimize when PK is deferrable explain (costs off) select * from t3 group by a,b,c; diff --git a/src/test/regress/expected/eager_aggregate.out b/src/test/regress/expected/eager_aggregate.out index 091ae48a92b..90578629061 100644 --- a/src/test/regress/expected/eager_aggregate.out +++ b/src/test/regress/expected/eager_aggregate.out @@ -34,14 +34,16 @@ GROUP BY t1.a ORDER BY t1.a; Hash Cond: (t1.b = t2.b) -> Seq Scan on public.eager_agg_t1 t1 Output: t1.a, t1.b, t1.c + Bloom Filter 1: keys=(t1.b) -> Hash Output: t2.b, (PARTIAL avg(t2.c)) + Bloom Filter 1 -> Partial HashAggregate Output: t2.b, PARTIAL avg(t2.c) Group Key: t2.b -> Seq Scan on public.eager_agg_t2 t2 Output: t2.a, t2.b, t2.c -(18 rows) +(20 rows) SELECT t1.a, avg(t2.c) FROM eager_agg_t1 t1 @@ -80,8 +82,10 @@ GROUP BY t1.a ORDER BY t1.a; Hash Cond: (t1.b = t2.b) -> Seq Scan on public.eager_agg_t1 t1 Output: t1.a, t1.b, t1.c + Bloom Filter 1: keys=(t1.b) -> Hash Output: t2.b, (PARTIAL avg(t2.c)) + Bloom Filter 1 -> Partial GroupAggregate Output: t2.b, PARTIAL avg(t2.c) Group Key: t2.b @@ -90,7 +94,7 @@ GROUP BY t1.a ORDER BY t1.a; Sort Key: t2.b -> Seq Scan on public.eager_agg_t2 t2 Output: t2.c, t2.b -(21 rows) +(23 rows) SELECT t1.a, avg(t2.c) FROM eager_agg_t1 t1 @@ -134,21 +138,25 @@ GROUP BY t1.a ORDER BY t1.a; Hash Cond: (t1.b = t2.b) -> Seq Scan on public.eager_agg_t1 t1 Output: t1.a, t1.b, t1.c + Bloom Filter 2: keys=(t1.b) -> Hash Output: t2.b, (PARTIAL avg((t2.c + t3.c))) + Bloom Filter 2 -> Partial HashAggregate Output: t2.b, PARTIAL avg((t2.c + t3.c)) Group Key: t2.b -> Hash Join Output: t2.c, t2.b, t3.c - Hash Cond: (t3.a = t2.a) - -> Seq Scan on public.eager_agg_t3 t3 - Output: t3.a, t3.b, t3.c + Hash Cond: (t2.a = t3.a) + -> Seq Scan on public.eager_agg_t2 t2 + Output: t2.a, t2.b, t2.c + Bloom Filter 1: keys=(t2.a) -> Hash - Output: t2.c, t2.b, t2.a - -> Seq Scan on public.eager_agg_t2 t2 - Output: t2.c, t2.b, t2.a -(25 rows) + Output: t3.c, t3.a + Bloom Filter 1 + -> Seq Scan on public.eager_agg_t3 t3 + Output: t3.c, t3.a +(29 rows) SELECT t1.a, avg(t2.c + t3.c) FROM eager_agg_t1 t1 @@ -189,8 +197,10 @@ GROUP BY t1.a ORDER BY t1.a; Hash Cond: (t1.b = t2.b) -> Seq Scan on public.eager_agg_t1 t1 Output: t1.a, t1.b, t1.c + Bloom Filter 2: keys=(t1.b) -> Hash Output: t2.b, (PARTIAL avg((t2.c + t3.c))) + Bloom Filter 2 -> Partial GroupAggregate Output: t2.b, PARTIAL avg((t2.c + t3.c)) Group Key: t2.b @@ -199,14 +209,16 @@ GROUP BY t1.a ORDER BY t1.a; Sort Key: t2.b -> Hash Join Output: t2.c, t2.b, t3.c - Hash Cond: (t3.a = t2.a) - -> Seq Scan on public.eager_agg_t3 t3 - Output: t3.a, t3.b, t3.c + Hash Cond: (t2.a = t3.a) + -> Seq Scan on public.eager_agg_t2 t2 + Output: t2.a, t2.b, t2.c + Bloom Filter 1: keys=(t2.a) -> Hash - Output: t2.c, t2.b, t2.a - -> Seq Scan on public.eager_agg_t2 t2 - Output: t2.c, t2.b, t2.a -(28 rows) + Output: t3.c, t3.a + Bloom Filter 1 + -> Seq Scan on public.eager_agg_t3 t3 + Output: t3.c, t3.a +(32 rows) SELECT t1.a, avg(t2.c + t3.c) FROM eager_agg_t1 t1 @@ -400,14 +412,16 @@ GROUP BY t1.a ORDER BY t1.a; Hash Cond: (t1.b = t2.b) -> Seq Scan on public.eager_agg_t1 t1 Output: t1.a, t1.b, t1.c + Bloom Filter 1: keys=(t1.b) -> Hash Output: t2.b, (PARTIAL avg(t2.c)) + Bloom Filter 1 -> Partial HashAggregate Output: t2.b, PARTIAL avg(t2.c) Group Key: t2.b -> Seq Scan on public.eager_agg_t2 t2 Output: t2.a, t2.b, t2.c -(18 rows) +(20 rows) SELECT t1.a, avg(t2.c) FROM eager_agg_t1 t1 @@ -442,11 +456,13 @@ GROUP BY t1.a ORDER BY t1.a; -> Sort Sort Key: t1.a -> Hash Join - Hash Cond: (t2.b = t1.b) - -> Seq Scan on eager_agg_t2 t2 + Hash Cond: (t1.b = t2.b) + -> Seq Scan on eager_agg_t1 t1 + Bloom Filter 1: keys=(b) -> Hash - -> Seq Scan on eager_agg_t1 t1 -(9 rows) + Bloom Filter 1 + -> Seq Scan on eager_agg_t2 t2 +(11 rows) EXPLAIN (COSTS OFF) SELECT t1.a, avg(t2.c) FILTER (WHERE random() > 0.5) @@ -460,11 +476,13 @@ GROUP BY t1.a ORDER BY t1.a; -> Sort Sort Key: t1.a -> Hash Join - Hash Cond: (t2.b = t1.b) - -> Seq Scan on eager_agg_t2 t2 + Hash Cond: (t1.b = t2.b) + -> Seq Scan on eager_agg_t1 t1 + Bloom Filter 1: keys=(b) -> Hash - -> Seq Scan on eager_agg_t1 t1 -(9 rows) + Bloom Filter 1 + -> Seq Scan on eager_agg_t2 t2 +(11 rows) -- Eager aggregation must not push a partial aggregate onto the inner side of a -- SEMI or ANTI join @@ -530,14 +548,16 @@ GROUP BY t2.b ORDER BY t2.b; Hash Cond: (t1.b = t2.b) -> Seq Scan on public.eager_agg_t1 t1 Output: t1.a, t1.b, t1.c + Bloom Filter 1: keys=(t1.b) -> Hash Output: t2.b, (PARTIAL count(*)) + Bloom Filter 1 -> Partial HashAggregate Output: t2.b, PARTIAL count(*) Group Key: t2.b -> Seq Scan on public.eager_agg_t2 t2 Output: t2.a, t2.b, t2.c -(18 rows) +(20 rows) SELECT t2.b, count(*) FROM eager_agg_t2 t2 diff --git a/src/test/regress/expected/graph_table.out b/src/test/regress/expected/graph_table.out index 46566b2e32f..6241b189ed8 100644 --- a/src/test/regress/expected/graph_table.out +++ b/src/test/regress/expected/graph_table.out @@ -259,8 +259,8 @@ SELECT * FROM x1, GRAPH_TABLE (myshop MATCH (c IS customers WHERE c.address = 'U SELECT x1.a, g.* FROM x1, GRAPH_TABLE (myshop MATCH (x1 IS customers WHERE x1.address = 'US')-[IS customer_orders]->(o IS orders) COLUMNS (x1.name AS customer_name, x1.customer_id AS cid, o.order_id)) g; a | customer_name | cid | order_id ---+---------------+-----+---------- - 1 | customer1 | 1 | 1 2 | customer1 | 1 | 1 + 1 | customer1 | 1 | 1 (2 rows) -- bare lateral reference in WHERE clause @@ -423,8 +423,8 @@ SELECT * FROM GRAPH_TABLE (g1 MATCH (a IS vl1 | vl2) COLUMNS (a.vname, a.vprop1) SELECT src, conn, dest, lprop1, vprop2, vprop1 FROM GRAPH_TABLE (g1 MATCH (a IS vl1)-[b IS el1]->(c IS vl2 | vl3) COLUMNS (a.vname AS src, b.ename AS conn, c.vname AS dest, c.lprop1, c.vprop2, c.vprop1)); src | conn | dest | lprop1 | vprop2 | vprop1 -----+------+------+----------+--------+-------- - v12 | e122 | v21 | vl2_prop | 1100 | 1010 v11 | e121 | v22 | vl2_prop | 1200 | 1020 + v12 | e122 | v21 | vl2_prop | 1100 | 1010 v11 | e131 | v33 | vl3_prop | | 2030 v11 | e132 | v31 | vl3_prop | | 2010 (4 rows) @@ -433,16 +433,16 @@ SELECT src, conn, dest, lprop1, vprop2, vprop1 FROM GRAPH_TABLE (g1 MATCH (a IS SELECT * FROM GRAPH_TABLE (g1 MATCH (v1 IS vl2)-[conn]-(v2) COLUMNS (v1.vname AS v1name, conn.ename AS cname, v2.vname AS v2name)); v1name | cname | v2name --------+-------+-------- - v21 | e122 | v12 v22 | e121 | v11 + v21 | e122 | v12 v22 | e231 | v32 (3 rows) SELECT * FROM GRAPH_TABLE (g1 MATCH (v1 IS vl2)-(v2) COLUMNS (v1.vname AS v1name, v2.vname AS v2name)); v1name | v2name --------+-------- - v21 | v12 v22 | v11 + v21 | v12 v22 | v32 (3 rows) @@ -503,8 +503,8 @@ LINE 1: SELECT * FROM GRAPH_TABLE (g1 MATCH (WHERE b.eprop1 = 10001)... SELECT * FROM GRAPH_TABLE (g1 MATCH (src)-[conn]->(dest) COLUMNS (src.vname AS svname, conn.ename AS cename, dest.vname AS dvname, src.vprop1 AS svp1, src.vprop2 AS svp2, src.lprop1 AS slp1, dest.vprop1 AS dvp1, dest.vprop2 AS dvp2, dest.lprop1 AS dlp1, conn.eprop1 AS cep1, conn.lprop2 AS clp2)); svname | cename | dvname | svp1 | svp2 | slp1 | dvp1 | dvp2 | dlp1 | cep1 | clp2 --------+--------+--------+------+------+----------+------+------+----------+-------+-------- - v12 | e122 | v21 | 20 | | | 1010 | 1100 | vl2_prop | 10002 | v11 | e121 | v22 | 10 | | | 1020 | 1200 | vl2_prop | 10001 | + v12 | e122 | v21 | 20 | | | 1010 | 1100 | vl2_prop | 10002 | v11 | e131 | v33 | 10 | | | 2030 | | vl3_prop | 10003 | v11 | e132 | v31 | 10 | | | 2010 | | vl3_prop | 10004 | v22 | e231 | v32 | 1020 | 1200 | vl2_prop | 2020 | | vl3_prop | | 100050 @@ -514,8 +514,8 @@ SELECT * FROM GRAPH_TABLE (g1 MATCH (src)-[conn]->(dest) COLUMNS (src.vname AS s SELECT * FROM GRAPH_TABLE (g1 MATCH (src IS vl1 | vl2 | vl3)-[conn]->(dest) COLUMNS (src.vname AS svname, conn.ename AS cename, dest.vname AS dvname)); svname | cename | dvname --------+--------+-------- - v12 | e122 | v21 v11 | e121 | v22 + v12 | e122 | v21 v11 | e131 | v33 v11 | e132 | v31 v22 | e231 | v32 @@ -535,8 +535,8 @@ SELECT vn FROM all_vertices EXCEPT (SELECT svn FROM all_connected_vertices UNION SELECT sn, cn, dn FROM GRAPH_TABLE (g1 MATCH (src IS l1)-[conn IS l1]->(dest IS l1) COLUMNS (src.elname AS sn, conn.elname AS cn, dest.elname AS dn)); sn | cn | dn -----+------+----- - v12 | e122 | v21 v11 | e121 | v22 + v12 | e122 | v21 v11 | e131 | v33 v11 | e132 | v31 v22 | e231 | v32 diff --git a/src/test/regress/expected/join.out b/src/test/regress/expected/join.out index 19e2cca548b..f1d4dbfbcff 100644 --- a/src/test/regress/expected/join.out +++ b/src/test/regress/expected/join.out @@ -218,13 +218,13 @@ SELECT t1.a, t2.e WHERE t1.a = t2.d; a | e ---+---- - 0 | 1 | -1 2 | 2 - 2 | 4 3 | -3 + 2 | 4 5 | -5 5 | -5 + 0 | (7 rows) -- @@ -1573,13 +1573,13 @@ SELECT * FROM J1_TBL INNER JOIN J2_TBL USING (i); i | j | t | k ---+---+-------+---- - 0 | | zero | 1 | 4 | one | -1 2 | 3 | two | 2 - 2 | 3 | two | 4 3 | 2 | three | -3 + 2 | 3 | two | 4 5 | 0 | five | -5 5 | 0 | five | -5 + 0 | | zero | (7 rows) -- Same as above, slightly different syntax @@ -1587,13 +1587,13 @@ SELECT * FROM J1_TBL JOIN J2_TBL USING (i); i | j | t | k ---+---+-------+---- - 0 | | zero | 1 | 4 | one | -1 2 | 3 | two | 2 - 2 | 3 | two | 4 3 | 2 | three | -3 + 2 | 3 | two | 4 5 | 0 | five | -5 5 | 0 | five | -5 + 0 | | zero | (7 rows) SELECT * @@ -1681,35 +1681,35 @@ SELECT * FROM J1_TBL NATURAL JOIN J2_TBL; i | j | t | k ---+---+-------+---- - 0 | | zero | 1 | 4 | one | -1 2 | 3 | two | 2 - 2 | 3 | two | 4 3 | 2 | three | -3 + 2 | 3 | two | 4 5 | 0 | five | -5 5 | 0 | five | -5 + 0 | | zero | (7 rows) SELECT * FROM J1_TBL t1 (a, b, c) NATURAL JOIN J2_TBL t2 (a, d); a | b | c | d ---+---+-------+---- - 0 | | zero | 1 | 4 | one | -1 2 | 3 | two | 2 - 2 | 3 | two | 4 3 | 2 | three | -3 + 2 | 3 | two | 4 5 | 0 | five | -5 5 | 0 | five | -5 + 0 | | zero | (7 rows) SELECT * FROM J1_TBL t1 (a, b, c) NATURAL JOIN J2_TBL t2 (d, a); a | b | c | d ---+---+------+--- - 0 | | zero | 2 | 3 | two | 2 4 | 1 | four | 2 + 0 | | zero | (3 rows) -- mismatch number of columns @@ -1718,13 +1718,13 @@ SELECT * FROM J1_TBL t1 (a, b) NATURAL JOIN J2_TBL t2 (a); a | b | t | k ---+---+-------+---- - 0 | | zero | 1 | 4 | one | -1 2 | 3 | two | 2 - 2 | 3 | two | 4 3 | 2 | three | -3 + 2 | 3 | two | 4 5 | 0 | five | -5 5 | 0 | five | -5 + 0 | | zero | (7 rows) -- @@ -1734,22 +1734,22 @@ SELECT * FROM J1_TBL JOIN J2_TBL ON (J1_TBL.i = J2_TBL.i); i | j | t | i | k ---+---+-------+---+---- - 0 | | zero | 0 | 1 | 4 | one | 1 | -1 2 | 3 | two | 2 | 2 - 2 | 3 | two | 2 | 4 3 | 2 | three | 3 | -3 + 2 | 3 | two | 2 | 4 5 | 0 | five | 5 | -5 5 | 0 | five | 5 | -5 + 0 | | zero | 0 | (7 rows) SELECT * FROM J1_TBL JOIN J2_TBL ON (J1_TBL.i = J2_TBL.k); i | j | t | i | k ---+---+------+---+--- - 0 | | zero | | 0 2 | 3 | two | 2 | 2 4 | 1 | four | 2 | 4 + 0 | | zero | | 0 (3 rows) -- @@ -1917,12 +1917,14 @@ where exists(select * from tenk1 c -> Hash Join Hash Cond: (b.tenthous = a.tenthous) -> Seq Scan on tenk1 b + Bloom Filter 1: keys=(tenthous) -> Hash + Bloom Filter 1 -> Seq Scan on tenk1 a Filter: (tenthous < 5000) -> Hash -> Seq Scan on tenk1 c -(11 rows) +(13 rows) -- -- More complicated constructs @@ -2378,9 +2380,11 @@ order by t1.unique1; Hash Cond: ((t1.two = t2.two) AND (t1.unique1 = (SubPlan expr_1))) -> Bitmap Heap Scan on tenk1 t1 Recheck Cond: (unique1 < 10) + Bloom Filter 1: keys=(two, unique1) -> Bitmap Index Scan on tenk1_unique1 Index Cond: (unique1 < 10) -> Hash + Bloom Filter 1 -> Bitmap Heap Scan on tenk1 t2 Recheck Cond: (unique1 < 10) -> Bitmap Index Scan on tenk1_unique1 @@ -2392,7 +2396,7 @@ order by t1.unique1; -> Limit -> Index Only Scan using tenk1_unique1 on tenk1 Index Cond: ((unique1 IS NOT NULL) AND (unique1 = t2.unique1)) -(20 rows) +(22 rows) -- Ensure we get the expected result select t1.unique1,t2.unique1 from tenk1 t1 @@ -3116,16 +3120,18 @@ set enable_memoize to off; explain (costs off) select count(*) from tenk1 a, tenk1 b where a.hundred = b.thousand and (b.fivethous % 10) < 10; - QUERY PLAN ------------------------------------------------------------- + QUERY PLAN +------------------------------------------------------------------ Aggregate -> Hash Join - Hash Cond: (a.hundred = b.thousand) - -> Index Only Scan using tenk1_hundred on tenk1 a + Hash Cond: (b.thousand = a.hundred) + -> Seq Scan on tenk1 b + Filter: ((fivethous % 10) < 10) + Bloom Filter 1: keys=(thousand) -> Hash - -> Seq Scan on tenk1 b - Filter: ((fivethous % 10) < 10) -(7 rows) + Bloom Filter 1 + -> Index Only Scan using tenk1_hundred on tenk1 a +(9 rows) select count(*) from tenk1 a, tenk1 b where a.hundred = b.thousand and (b.fivethous % 10) < 10; @@ -3286,14 +3292,14 @@ select 1 from tenk1 where (hundred, thousand) in (select twothousand, twothousand from onek); QUERY PLAN ------------------------------------------------- - Hash Join - Hash Cond: (tenk1.hundred = onek.twothousand) - -> Seq Scan on tenk1 - Filter: (hundred = thousand) + Hash Right Semi Join + Hash Cond: (onek.twothousand = tenk1.hundred) + -> Seq Scan on onek + Bloom Filter 1: keys=(twothousand) -> Hash - -> HashAggregate - Group Key: onek.twothousand - -> Seq Scan on onek + Bloom Filter 1 + -> Seq Scan on tenk1 + Filter: (hundred = thousand) (8 rows) reset enable_memoize; @@ -3542,14 +3548,16 @@ create temp table tidv (idv mycomptype); create index on tidv (idv); explain (costs off) select a.idv, b.idv from tidv a, tidv b where a.idv = b.idv; - QUERY PLAN ----------------------------------------------------------- - Merge Join - Merge Cond: (a.idv = b.idv) - -> Index Only Scan using tidv_idv_idx on tidv a - -> Materialize - -> Index Only Scan using tidv_idv_idx on tidv b -(5 rows) + QUERY PLAN +------------------------------------ + Hash Join + Hash Cond: (a.idv = b.idv) + -> Seq Scan on tidv a + Bloom Filter 1: keys=(idv) + -> Hash + Bloom Filter 1 + -> Seq Scan on tidv b +(7 rows) set enable_mergejoin = 0; set enable_hashjoin = 0; @@ -5955,18 +5963,20 @@ explain (costs off) select a.unique1, b.unique2 from onek a left join onek b on a.unique1 = b.unique2 where (b.unique2, random() > 0) = any (select q1, random() > 0 from int8_tbl c where c.q1 < b.unique1); - QUERY PLAN --------------------------------------------------------------------------------------------------------------------------- + QUERY PLAN +-------------------------------------------------------------------------------------------------------------------------------- Hash Join - Hash Cond: (b.unique2 = a.unique1) - -> Seq Scan on onek b - Filter: (ANY ((unique2 = (SubPlan any_1).col1) AND ((random() > '0'::double precision) = (SubPlan any_1).col2))) - SubPlan any_1 - -> Seq Scan on int8_tbl c - Filter: (q1 < b.unique1) + Hash Cond: (a.unique1 = b.unique2) + -> Index Only Scan using onek_unique1 on onek a + Bloom Filter 1: keys=(unique1) -> Hash - -> Index Only Scan using onek_unique1 on onek a -(9 rows) + Bloom Filter 1 + -> Seq Scan on onek b + Filter: (ANY ((unique2 = (SubPlan any_1).col1) AND ((random() > '0'::double precision) = (SubPlan any_1).col2))) + SubPlan any_1 + -> Seq Scan on int8_tbl c + Filter: (q1 < b.unique1) +(11 rows) select a.unique1, b.unique2 from onek a left join onek b on a.unique1 = b.unique2 @@ -6922,23 +6932,27 @@ where exists (select 1 from t t4 Hash Cond: (t6.b = t4.b) -> Seq Scan on pg_temp.t t6 Output: t6.a, t6.b + Bloom Filter 2: keys=(t6.b) -> Hash Output: t4.b, t5.b, t5.a + Bloom Filter 2 -> Hash Join Output: t4.b, t5.b, t5.a Inner Unique: true Hash Cond: (t5.b = t4.b) -> Seq Scan on pg_temp.t t5 Output: t5.a, t5.b + Bloom Filter 1: keys=(t5.b) -> Hash Output: t4.b, t4.a + Bloom Filter 1 -> Index Scan using t_a_key on pg_temp.t t4 Output: t4.b, t4.a Index Cond: (t4.a = 1) -> Index Only Scan using t_a_key on pg_temp.t t3 Output: t3.a Index Cond: (t3.a = t5.a) -(32 rows) +(36 rows) select t1.a from t t1 left join t t2 on t1.a = t2.a @@ -8235,33 +8249,32 @@ JOIN ( ) ) _t2t3t4 ON sj_t1.id = _t2t3t4.id; - QUERY PLAN -------------------------------------------------------------------------------------- + QUERY PLAN +------------------------------------------------------------------------------------------- Nested Loop - Join Filter: (sj_t3.id = sj_t1.id) + Join Filter: (sj_t1.id = sj_t3.id) -> Nested Loop - Join Filter: (sj_t2.id = sj_t3.id) - -> Nested Loop Semi Join + Join Filter: (sj_t3.id = sj_t2_1.id) + -> Nested Loop + Join Filter: (sj_t2.id = sj_t3.id) -> Nested Loop - -> HashAggregate - Group Key: sj_t3.id + -> Unique + -> Nested Loop + -> Index Only Scan using sj_t3_a_id_idx on sj_t3 sj_t3_1 + Index Cond: (a = 1) + -> Seq Scan on sj_t4 sj_t4_1 + -> Index Only Scan using sj_t2_id_idx on sj_t2 + Index Cond: (id = sj_t3_1.id) + -> Materialize + -> Unique -> Nested Loop + -> Index Only Scan using sj_t3_a_id_idx on sj_t3 + Index Cond: (a = 1) -> Seq Scan on sj_t4 - -> Materialize - -> Bitmap Heap Scan on sj_t3 - Recheck Cond: (a = 1) - -> Bitmap Index Scan on sj_t3_a_id_idx - Index Cond: (a = 1) - -> Index Only Scan using sj_t2_id_idx on sj_t2 sj_t2_1 - Index Cond: (id = sj_t3.id) - -> Nested Loop - -> Index Only Scan using sj_t3_a_id_idx on sj_t3 sj_t3_1 - Index Cond: ((a = 1) AND (id = sj_t3.id)) - -> Seq Scan on sj_t4 sj_t4_1 - -> Index Only Scan using sj_t2_id_idx on sj_t2 - Index Cond: (id = sj_t2_1.id) + -> Index Only Scan using sj_t2_id_idx on sj_t2 sj_t2_1 + Index Cond: (id = sj_t2.id) -> Seq Scan on sj_t1 -(24 rows) +(23 rows) -- -- Test RowMarks-related code @@ -9557,12 +9570,14 @@ select * from fkest f1 Hash Cond: ((f2.x = f1.x) AND (f2.x10b = f1.x10)) -> Seq Scan on fkest f2 Filter: (x100 = 2) + Bloom Filter 1: keys=(x, x10b) -> Hash + Bloom Filter 1 -> Seq Scan on fkest f1 Filter: (x100 = 2) -> Index Scan using fkest_x_x10_x100_idx on fkest f3 Index Cond: (x = f1.x) -(10 rows) +(12 rows) alter table fkest add constraint fk foreign key (x, x10b, x100) references fkest (x, x10, x100); @@ -9571,20 +9586,21 @@ select * from fkest f1 join fkest f2 on (f1.x = f2.x and f1.x10 = f2.x10b and f1.x100 = f2.x100) join fkest f3 on f1.x = f3.x where f1.x100 = 2; - QUERY PLAN ------------------------------------------------------ + QUERY PLAN +--------------------------------------------------------------- Hash Join Hash Cond: ((f2.x = f1.x) AND (f2.x10b = f1.x10)) - -> Hash Join - Hash Cond: (f3.x = f2.x) - -> Seq Scan on fkest f3 - -> Hash - -> Seq Scan on fkest f2 - Filter: (x100 = 2) + -> Nested Loop + -> Seq Scan on fkest f2 + Filter: (x100 = 2) + Bloom Filter 1: keys=(x, x10b) + -> Index Scan using fkest_x_x10_x100_idx on fkest f3 + Index Cond: (x = f2.x) -> Hash + Bloom Filter 1 -> Seq Scan on fkest f1 Filter: (x100 = 2) -(11 rows) +(12 rows) rollback; -- @@ -9670,19 +9686,20 @@ select * from j1 inner join j2 on j1.id > j2.id; -- ensure non-unique rel is not chosen as inner explain (verbose, costs off) select * from j1 inner join j3 on j1.id = j3.id; - QUERY PLAN ------------------------------------ + QUERY PLAN +-------------------------------------- Hash Join Output: j1.id, j3.id - Inner Unique: true - Hash Cond: (j3.id = j1.id) - -> Seq Scan on public.j3 - Output: j3.id - -> Hash + Hash Cond: (j1.id = j3.id) + -> Seq Scan on public.j1 Output: j1.id - -> Seq Scan on public.j1 - Output: j1.id -(10 rows) + Bloom Filter 1: keys=(j1.id) + -> Hash + Output: j3.id + Bloom Filter 1 + -> Seq Scan on public.j3 + Output: j3.id +(11 rows) -- ensure left join is marked as unique explain (verbose, costs off) @@ -9771,44 +9788,52 @@ select * from j1 natural join j2; explain (verbose, costs off) select * from j1 inner join (select distinct id from j3) j3 on j1.id = j3.id; - QUERY PLAN ------------------------------------------ - Nested Loop + QUERY PLAN +----------------------------------------------- + Hash Join Output: j1.id, j3.id Inner Unique: true - Join Filter: (j1.id = j3.id) - -> Unique + Hash Cond: (j1.id = j3.id) + -> Seq Scan on public.j1 + Output: j1.id + Bloom Filter 1: keys=(j1.id) + -> Hash Output: j3.id - -> Sort + Bloom Filter 1 + -> Unique Output: j3.id - Sort Key: j3.id - -> Seq Scan on public.j3 + -> Sort Output: j3.id - -> Seq Scan on public.j1 - Output: j1.id -(13 rows) + Sort Key: j3.id + -> Seq Scan on public.j3 + Output: j3.id +(17 rows) -- ensure group by clause allows the inner to become unique explain (verbose, costs off) select * from j1 inner join (select id from j3 group by id) j3 on j1.id = j3.id; - QUERY PLAN ------------------------------------------ - Nested Loop + QUERY PLAN +----------------------------------------------- + Hash Join Output: j1.id, j3.id Inner Unique: true - Join Filter: (j1.id = j3.id) - -> Group + Hash Cond: (j1.id = j3.id) + -> Seq Scan on public.j1 + Output: j1.id + Bloom Filter 1: keys=(j1.id) + -> Hash Output: j3.id - Group Key: j3.id - -> Sort + Bloom Filter 1 + -> Group Output: j3.id - Sort Key: j3.id - -> Seq Scan on public.j3 + Group Key: j3.id + -> Sort Output: j3.id - -> Seq Scan on public.j1 - Output: j1.id -(14 rows) + Sort Key: j3.id + -> Seq Scan on public.j3 + Output: j3.id +(18 rows) drop table j1; drop table j2; @@ -9922,14 +9947,16 @@ create index j2_id1_idx on j2 (id1) where id1 % 1000 = 1; explain (costs off) select * from j1 inner join j2 on j1.id1 = j2.id1 and j1.id2 = j2.id2 where j1.id1 % 1000 = 1 and j2.id1 % 1000 = 1; - QUERY PLAN ------------------------------------------ - Merge Join - Merge Cond: (j1.id1 = j2.id1) - Join Filter: (j2.id2 = j1.id2) - -> Index Scan using j1_id1_idx on j1 - -> Index Scan using j2_id1_idx on j2 -(5 rows) + QUERY PLAN +---------------------------------------------------------- + Nested Loop + Disabled: true + Join Filter: ((j2.id1 = j1.id1) AND (j2.id2 = j1.id2)) + -> Seq Scan on j1 + Filter: ((id1 % 1000) = 1) + -> Seq Scan on j2 + Filter: ((id1 % 1000) = 1) +(7 rows) select * from j1 inner join j2 on j1.id1 = j2.id1 and j1.id2 = j2.id2 @@ -9944,15 +9971,16 @@ where j1.id1 % 1000 = 1 and j2.id1 % 1000 = 1; explain (costs off) select * from j1 inner join j2 on j1.id1 = j2.id1 and j1.id2 = j2.id2 where j1.id1 % 1000 = 1 and j2.id1 % 1000 = 1 and j2.id1 = any (array[1]); - QUERY PLAN ----------------------------------------------------- - Merge Join - Merge Cond: (j1.id1 = j2.id1) - Join Filter: (j2.id2 = j1.id2) - -> Index Scan using j1_id1_idx on j1 - -> Index Scan using j2_id1_idx on j2 - Index Cond: (id1 = ANY ('{1}'::integer[])) -(6 rows) + QUERY PLAN +------------------------------------------------------------------------- + Nested Loop + Disabled: true + Join Filter: ((j2.id1 = j1.id1) AND (j2.id2 = j1.id2)) + -> Seq Scan on j1 + Filter: ((id1 % 1000) = 1) + -> Seq Scan on j2 + Filter: ((id1 = ANY ('{1}'::integer[])) AND ((id1 % 1000) = 1)) +(7 rows) select * from j1 inner join j2 on j1.id1 = j2.id1 and j1.id2 = j2.id2 @@ -9967,15 +9995,16 @@ where j1.id1 % 1000 = 1 and j2.id1 % 1000 = 1 and j2.id1 = any (array[1]); explain (costs off) select * from j1 inner join j2 on j1.id1 = j2.id1 and j1.id2 = j2.id2 where j1.id1 % 1000 = 1 and j2.id1 % 1000 = 1 and j2.id1 >= any (array[1,5]); - QUERY PLAN -------------------------------------------------------- - Merge Join - Merge Cond: (j1.id1 = j2.id1) - Join Filter: (j2.id2 = j1.id2) - -> Index Scan using j1_id1_idx on j1 - -> Index Scan using j2_id1_idx on j2 - Index Cond: (id1 >= ANY ('{1,5}'::integer[])) -(6 rows) + QUERY PLAN +---------------------------------------------------------------------------- + Nested Loop + Disabled: true + Join Filter: ((j2.id1 = j1.id1) AND (j2.id2 = j1.id2)) + -> Seq Scan on j1 + Filter: ((id1 % 1000) = 1) + -> Seq Scan on j2 + Filter: ((id1 >= ANY ('{1,5}'::integer[])) AND ((id1 % 1000) = 1)) +(7 rows) select * from j1 inner join j2 on j1.id1 = j2.id1 and j1.id2 = j2.id2 diff --git a/src/test/regress/expected/join_hash.out b/src/test/regress/expected/join_hash.out index 75009e29720..21900564149 100644 --- a/src/test/regress/expected/join_hash.out +++ b/src/test/regress/expected/join_hash.out @@ -330,9 +330,11 @@ explain (costs off) -> Hash Join Hash Cond: (r.id = s.id) -> Seq Scan on simple r + Bloom Filter 1: keys=(id) -> Hash + Bloom Filter 1 -> Seq Scan on bigger_than_it_looks s -(6 rows) +(8 rows) select count(*) FROM simple r JOIN bigger_than_it_looks s USING (id); count @@ -445,9 +447,11 @@ explain (costs off) -> Hash Join Hash Cond: (r.id = s.id) -> Seq Scan on simple r + Bloom Filter 1: keys=(id) -> Hash + Bloom Filter 1 -> Seq Scan on extremely_skewed s -(6 rows) +(8 rows) select count(*) from simple r join extremely_skewed s using (id); count @@ -1149,9 +1153,11 @@ lateral (select t1.fivethous, i4.f1 from tenk1 t1 join int4_tbl i4 -> Hash Join Hash Cond: (t1.fivethous = (i4.f1 + i8.q2)) -> Seq Scan on tenk1 t1 + Bloom Filter 1: keys=(fivethous) -> Hash + Bloom Filter 1 -> Seq Scan on int4_tbl i4 -(9 rows) +(11 rows) select i8.q2, ss.* from int8_tbl i8, diff --git a/src/test/regress/expected/merge.out b/src/test/regress/expected/merge.out index 9cb1d87066a..40461acf17c 100644 --- a/src/test/regress/expected/merge.out +++ b/src/test/regress/expected/merge.out @@ -39,18 +39,17 @@ USING source AS s ON t.tid = s.sid WHEN MATCHED THEN DELETE; - QUERY PLAN ----------------------------------------- + QUERY PLAN +------------------------------------------ Merge on target t - -> Merge Join - Merge Cond: (t.tid = s.sid) - -> Sort - Sort Key: t.tid - -> Seq Scan on target t - -> Sort - Sort Key: s.sid + -> Hash Join + Hash Cond: (t.tid = s.sid) + -> Seq Scan on target t + Bloom Filter 1: keys=(tid) + -> Hash + Bloom Filter 1 -> Seq Scan on source s -(9 rows) +(8 rows) -- -- Errors @@ -323,15 +322,17 @@ USING source AS s ON t.tid = s.sid WHEN MATCHED THEN UPDATE SET balance = 0; - QUERY PLAN ----------------------------------------- + QUERY PLAN +------------------------------------------ Merge on target t -> Hash Join Hash Cond: (s.sid = t.tid) -> Seq Scan on source s + Bloom Filter 1: keys=(sid) -> Hash + Bloom Filter 1 -> Seq Scan on target t -(6 rows) +(8 rows) EXPLAIN (COSTS OFF) MERGE INTO target t @@ -339,15 +340,17 @@ USING source AS s ON t.tid = s.sid WHEN MATCHED THEN DELETE; - QUERY PLAN ----------------------------------------- + QUERY PLAN +------------------------------------------ Merge on target t -> Hash Join Hash Cond: (s.sid = t.tid) -> Seq Scan on source s + Bloom Filter 1: keys=(sid) -> Hash + Bloom Filter 1 -> Seq Scan on target t -(6 rows) +(8 rows) EXPLAIN (COSTS OFF) MERGE INTO target t @@ -1636,42 +1639,38 @@ SELECT explain_merge(' MERGE INTO ex_mtarget t USING ex_msource s ON t.a = s.a WHEN MATCHED THEN UPDATE SET b = t.b + 1'); - explain_merge -------------------------------------------------------------------------- + explain_merge +------------------------------------------------------------------------------------------ Merge on ex_mtarget t (actual rows=0.00 loops=1) Tuples: updated=50 - -> Merge Join (actual rows=50.00 loops=1) - Merge Cond: (t.a = s.a) - -> Sort (actual rows=50.00 loops=1) - Sort Key: t.a - Sort Method: quicksort Memory: xxx - -> Seq Scan on ex_mtarget t (actual rows=50.00 loops=1) - -> Sort (actual rows=100.00 loops=1) - Sort Key: s.a - Sort Method: quicksort Memory: xxx + -> Hash Join (actual rows=50.00 loops=1) + Hash Cond: (t.a = s.a) + -> Seq Scan on ex_mtarget t (actual rows=50.00 loops=1) + Bloom Filter 1: keys=(a) checked=49 rejected=0 (0.0%) + -> Hash (actual rows=100.00 loops=1) + Buckets: xxx Batches: xxx Memory Usage: xxx + Bloom Filter 1: bits=8388608 hashes=10 memory=1024kB checked=49 rejected=0 -> Seq Scan on ex_msource s (actual rows=100.00 loops=1) -(12 rows) +(10 rows) -- only updates to selected tuples SELECT explain_merge(' MERGE INTO ex_mtarget t USING ex_msource s ON t.a = s.a WHEN MATCHED AND t.a < 10 THEN UPDATE SET b = t.b + 1'); - explain_merge -------------------------------------------------------------------------- + explain_merge +------------------------------------------------------------------------------------------ Merge on ex_mtarget t (actual rows=0.00 loops=1) Tuples: updated=5 skipped=45 - -> Merge Join (actual rows=50.00 loops=1) - Merge Cond: (t.a = s.a) - -> Sort (actual rows=50.00 loops=1) - Sort Key: t.a - Sort Method: quicksort Memory: xxx - -> Seq Scan on ex_mtarget t (actual rows=50.00 loops=1) - -> Sort (actual rows=100.00 loops=1) - Sort Key: s.a - Sort Method: quicksort Memory: xxx + -> Hash Join (actual rows=50.00 loops=1) + Hash Cond: (t.a = s.a) + -> Seq Scan on ex_mtarget t (actual rows=50.00 loops=1) + Bloom Filter 1: keys=(a) checked=49 rejected=0 (0.0%) + -> Hash (actual rows=100.00 loops=1) + Buckets: xxx Batches: xxx Memory Usage: xxx + Bloom Filter 1: bits=8388608 hashes=10 memory=1024kB checked=49 rejected=0 -> Seq Scan on ex_msource s (actual rows=100.00 loops=1) -(12 rows) +(10 rows) -- updates + deletes SELECT explain_merge(' @@ -1680,21 +1679,19 @@ WHEN MATCHED AND t.a < 10 THEN UPDATE SET b = t.b + 1 WHEN MATCHED AND t.a >= 10 AND t.a <= 20 THEN DELETE'); - explain_merge -------------------------------------------------------------------------- + explain_merge +------------------------------------------------------------------------------------------ Merge on ex_mtarget t (actual rows=0.00 loops=1) Tuples: updated=5 deleted=5 skipped=40 - -> Merge Join (actual rows=50.00 loops=1) - Merge Cond: (t.a = s.a) - -> Sort (actual rows=50.00 loops=1) - Sort Key: t.a - Sort Method: quicksort Memory: xxx - -> Seq Scan on ex_mtarget t (actual rows=50.00 loops=1) - -> Sort (actual rows=100.00 loops=1) - Sort Key: s.a - Sort Method: quicksort Memory: xxx + -> Hash Join (actual rows=50.00 loops=1) + Hash Cond: (t.a = s.a) + -> Seq Scan on ex_mtarget t (actual rows=50.00 loops=1) + Bloom Filter 1: keys=(a) checked=49 rejected=0 (0.0%) + -> Hash (actual rows=100.00 loops=1) + Buckets: xxx Batches: xxx Memory Usage: xxx + Bloom Filter 1: bits=8388608 hashes=10 memory=1024kB checked=49 rejected=0 -> Seq Scan on ex_msource s (actual rows=100.00 loops=1) -(12 rows) +(10 rows) -- only inserts SELECT explain_merge(' @@ -1791,21 +1788,20 @@ SELECT explain_merge(' MERGE INTO ex_mtarget t USING ex_msource s ON t.a = s.a AND t.a < -1000 WHEN MATCHED AND t.a < 10 THEN DO NOTHING'); - explain_merge ------------------------------------------------------------------------ + explain_merge +----------------------------------------------------------------------------------------- Merge on ex_mtarget t (actual rows=0.00 loops=1) - -> Merge Join (actual rows=0.00 loops=1) - Merge Cond: (t.a = s.a) - -> Sort (actual rows=0.00 loops=1) - Sort Key: t.a - Sort Method: quicksort Memory: xxx + -> Hash Join (actual rows=0.00 loops=1) + Hash Cond: (s.a = t.a) + -> Seq Scan on ex_msource s (actual rows=1.00 loops=1) + Bloom Filter 1: keys=(a) checked=0 rejected=0 (0.0%) + -> Hash (actual rows=0.00 loops=1) + Buckets: xxx Batches: xxx Memory Usage: xxx + Bloom Filter 1: bits=8388608 hashes=10 memory=1024kB checked=0 rejected=0 -> Seq Scan on ex_mtarget t (actual rows=0.00 loops=1) Filter: (a < '-1000'::integer) Rows Removed by Filter: 54 - -> Sort (never executed) - Sort Key: s.a - -> Seq Scan on ex_msource s (never executed) -(12 rows) +(11 rows) DROP TABLE ex_msource, ex_mtarget; DROP FUNCTION explain_merge(text); @@ -1831,8 +1827,10 @@ WHEN MATCHED AND t.c > s.cnt THEN Join Filter: (t.b < (SubPlan expr_1)) -> Seq Scan on public.tgt t Output: t.ctid, t.a, t.b + Bloom Filter 1: keys=(t.a) -> Hash Output: s.a, s.b, s.c, s.d, s.ctid + Bloom Filter 1 -> Seq Scan on public.src s Output: s.a, s.b, s.c, s.d, s.ctid SubPlan expr_1 @@ -1856,7 +1854,7 @@ WHEN MATCHED AND t.c > s.cnt THEN -> Seq Scan on public.ref r_1 Output: r_1.ab, r_1.cd Filter: ((r_1.ab = (s.a + s.b)) AND (r_1.cd = (s.c - s.d))) -(32 rows) +(34 rows) DROP TABLE src, tgt, ref; -- Subqueries diff --git a/src/test/regress/expected/misc_functions.out b/src/test/regress/expected/misc_functions.out index c3261bff209..7fff6a720aa 100644 --- a/src/test/regress/expected/misc_functions.out +++ b/src/test/regress/expected/misc_functions.out @@ -582,10 +582,12 @@ WHERE my_int_eq(a.unique2, 42); Hash Join Hash Cond: (b.unique1 = a.unique1) -> Seq Scan on tenk1 b + Bloom Filter 1: keys=(unique1) -> Hash + Bloom Filter 1 -> Seq Scan on tenk1 a Filter: my_int_eq(unique2, 42) -(6 rows) +(8 rows) -- With support function that knows it's int4eq, we get a different plan CREATE FUNCTION test_support_func(internal) @@ -612,14 +614,16 @@ CREATE FUNCTION my_gen_series(int, int) RETURNS SETOF integer SUPPORT test_support_func; EXPLAIN (COSTS OFF) SELECT * FROM tenk1 a JOIN my_gen_series(1,1000) g ON a.unique1 = g; - QUERY PLAN ----------------------------------------- + QUERY PLAN +---------------------------------------------- Hash Join - Hash Cond: (g.g = a.unique1) - -> Function Scan on my_gen_series g + Hash Cond: (a.unique1 = g.g) + -> Seq Scan on tenk1 a + Bloom Filter 1: keys=(unique1) -> Hash - -> Seq Scan on tenk1 a -(5 rows) + Bloom Filter 1 + -> Function Scan on my_gen_series g +(7 rows) EXPLAIN (COSTS OFF) SELECT * FROM tenk1 a JOIN my_gen_series(1,10) g ON a.unique1 = g; diff --git a/src/test/regress/expected/privileges.out b/src/test/regress/expected/privileges.out index f6cc1a1029c..cb03966e79b 100644 --- a/src/test/regress/expected/privileges.out +++ b/src/test/regress/expected/privileges.out @@ -618,10 +618,12 @@ EXPLAIN (COSTS OFF) SELECT * FROM atest12 x, atest12 y Hash Join Hash Cond: (x.a = y.b) -> Seq Scan on atest12 x + Bloom Filter 1: keys=(a) -> Hash + Bloom Filter 1 -> Seq Scan on atest12 y Filter: (abs(a) <<< 5) -(6 rows) +(8 rows) -- clean up (regress_priv_user1's objects are all dropped later) DROP FUNCTION leak2(integer, integer) CASCADE; diff --git a/src/test/regress/expected/returning.out b/src/test/regress/expected/returning.out index ca83d9fcc09..50cd3be8030 100644 --- a/src/test/regress/expected/returning.out +++ b/src/test/regress/expected/returning.out @@ -713,29 +713,31 @@ UPDATE joinview SET f3 = f3 + 1 WHERE f3 = 57 Update on pg_temp.foo foo_1 -> Hash Join Output: foo_2.f1, (foo_2.f3 + 1), joinme.ctid, foo_2.ctid, joinme_1.ctid, joinme.other, foo_1.tableoid, foo_1.ctid, foo_2.tableoid - Hash Cond: (foo_1.f2 = joinme.f2j) - -> Hash Join - Output: foo_1.f2, foo_1.tableoid, foo_1.ctid, joinme_1.ctid, joinme_1.f2j - Hash Cond: (joinme_1.f2j = foo_1.f2) - -> Seq Scan on pg_temp.joinme joinme_1 - Output: joinme_1.ctid, joinme_1.f2j - -> Hash - Output: foo_1.f2, foo_1.tableoid, foo_1.ctid - -> Seq Scan on pg_temp.foo foo_1 - Output: foo_1.f2, foo_1.tableoid, foo_1.ctid + Hash Cond: (joinme_1.f2j = foo_1.f2) + -> Seq Scan on pg_temp.joinme joinme_1 + Output: joinme_1.ctid, joinme_1.f2j + Bloom Filter 2: keys=(joinme_1.f2j) -> Hash - Output: joinme.ctid, joinme.other, joinme.f2j, foo_2.f1, foo_2.f3, foo_2.ctid, foo_2.f2, foo_2.tableoid + Output: foo_1.f2, foo_1.tableoid, foo_1.ctid, joinme.ctid, joinme.other, joinme.f2j, foo_2.f1, foo_2.f3, foo_2.ctid, foo_2.f2, foo_2.tableoid + Bloom Filter 2 -> Hash Join - Output: joinme.ctid, joinme.other, joinme.f2j, foo_2.f1, foo_2.f3, foo_2.ctid, foo_2.f2, foo_2.tableoid - Hash Cond: (joinme.f2j = foo_2.f2) + Output: foo_1.f2, foo_1.tableoid, foo_1.ctid, joinme.ctid, joinme.other, joinme.f2j, foo_2.f1, foo_2.f3, foo_2.ctid, foo_2.f2, foo_2.tableoid + Hash Cond: (joinme.f2j = foo_1.f2) -> Seq Scan on pg_temp.joinme Output: joinme.ctid, joinme.other, joinme.f2j + Bloom Filter 1: keys=(joinme.f2j) -> Hash - Output: foo_2.f1, foo_2.f3, foo_2.ctid, foo_2.f2, foo_2.tableoid - -> Seq Scan on pg_temp.foo foo_2 - Output: foo_2.f1, foo_2.f3, foo_2.ctid, foo_2.f2, foo_2.tableoid - Filter: (foo_2.f3 = 57) -(27 rows) + Output: foo_1.f2, foo_1.tableoid, foo_1.ctid, foo_2.f1, foo_2.f3, foo_2.ctid, foo_2.f2, foo_2.tableoid + Bloom Filter 1 + -> Nested Loop + Output: foo_1.f2, foo_1.tableoid, foo_1.ctid, foo_2.f1, foo_2.f3, foo_2.ctid, foo_2.f2, foo_2.tableoid + Join Filter: (foo_1.f2 = foo_2.f2) + -> Seq Scan on pg_temp.foo foo_2 + Output: foo_2.f1, foo_2.f3, foo_2.ctid, foo_2.f2, foo_2.tableoid + Filter: (foo_2.f3 = 57) + -> Seq Scan on pg_temp.foo foo_1 + Output: foo_1.f2, foo_1.tableoid, foo_1.ctid +(29 rows) UPDATE joinview SET f3 = f3 + 1 WHERE f3 = 57 RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3; @@ -793,12 +795,14 @@ UPDATE joinview SET f3 = f3 + 1, f4 = 7 WHERE f3 = 58 Hash Cond: (joinme.f2j = foo.f2) -> Seq Scan on pg_temp.joinme Output: joinme.other, joinme.ctid, joinme.f2j + Bloom Filter 1: keys=(joinme.f2j) -> Hash Output: foo.f3, foo.f1, foo.f2, foo.f4, foo.ctid, foo.tableoid + Bloom Filter 1 -> Seq Scan on pg_temp.foo Output: foo.f3, foo.f1, foo.f2, foo.f4, foo.ctid, foo.tableoid Filter: (foo.f3 = 58) -(12 rows) +(14 rows) UPDATE joinview SET f3 = f3 + 1, f4 = 7 WHERE f3 = 58 RETURNING old.*, new.*, *, new.f3 - old.f3 AS delta_f3; -- should succeed diff --git a/src/test/regress/expected/rowsecurity.out b/src/test/regress/expected/rowsecurity.out index 3a5e82c35bd..4d69d1cd84b 100644 --- a/src/test/regress/expected/rowsecurity.out +++ b/src/test/regress/expected/rowsecurity.out @@ -1,6 +1,8 @@ -- -- Test of Row-level security feature -- +-- disable bloom filter pushdown, to not interfere with calls to functions +SET enable_hashjoin_bloom = off; -- Clean up in case a prior regression run failed -- Suppress NOTICE messages when users/groups don't exist SET client_min_messages TO 'warning'; diff --git a/src/test/regress/expected/select_views.out b/src/test/regress/expected/select_views.out index 1aeed8452bd..fe7be9891d5 100644 --- a/src/test/regress/expected/select_views.out +++ b/src/test/regress/expected/select_views.out @@ -2,6 +2,8 @@ -- SELECT_VIEWS -- test the views defined in CREATE_VIEWS -- +-- disable bloom filter pushdown, to not interfere with calls to functions +SET enable_hashjoin_bloom = off; SELECT * FROM street; name | thepath | cname ------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------- diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out index 37070c1a896..07854247020 100644 --- a/src/test/regress/expected/stats_ext.out +++ b/src/test/regress/expected/stats_ext.out @@ -3610,17 +3610,16 @@ ANALYZE sb_1, sb_2; -- bucket size is quite big because there are possibly many correlations. EXPLAIN (COSTS OFF) -- Choose merge join SELECT * FROM sb_1 a, sb_2 b WHERE a.x = b.x AND a.y = b.y AND a.z = b.z; - QUERY PLAN -------------------------------------------------------------- - Merge Join - Merge Cond: ((a.z = b.z) AND (a.x = b.x) AND (a.y = b.y)) - -> Sort - Sort Key: a.z, a.x, a.y + QUERY PLAN +------------------------------------------------------------ + Hash Join + Hash Cond: ((b.x = a.x) AND (b.y = a.y) AND (b.z = a.z)) + -> Seq Scan on sb_2 b + Bloom Filter 1: keys=(x, y, z) + -> Hash + Bloom Filter 1 -> Seq Scan on sb_1 a - -> Sort - Sort Key: b.z, b.x, b.y - -> Seq Scan on sb_2 b -(8 rows) +(7 rows) -- The ndistinct extended statistics on (x, y, z) provides more reliable value -- of bucket size. diff --git a/src/test/regress/expected/subselect.out b/src/test/regress/expected/subselect.out index e7ff7191082..5f1f3c7996a 100644 --- a/src/test/regress/expected/subselect.out +++ b/src/test/regress/expected/subselect.out @@ -783,14 +783,16 @@ order by t1.a, t2.a; Hash Cond: (t2.a = (t3.b + 1)) -> Seq Scan on public.semijoin_unique_tbl t2 Output: t2.a, t2.b + Bloom Filter 1: keys=(t2.a) -> Hash Output: t3.a, t3.b + Bloom Filter 1 -> HashAggregate Output: t3.a, t3.b Group Key: (t3.a + 1), (t3.b + 1) -> Seq Scan on public.semijoin_unique_tbl t3 Output: t3.a, t3.b, (t3.a + 1), (t3.b + 1) -(24 rows) +(26 rows) -- encourage use of parallel plans set parallel_setup_cost=0; @@ -1669,7 +1671,7 @@ select * from int4_tbl where --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Nested Loop Semi Join Output: int4_tbl.f1 - Join Filter: (CASE WHEN (ANY (int4_tbl.f1 = (hashed SubPlan any_1).col1)) THEN int4_tbl.f1 ELSE NULL::integer END = b.ten) + Join Filter: (b.ten = CASE WHEN (ANY (int4_tbl.f1 = (hashed SubPlan any_1).col1)) THEN int4_tbl.f1 ELSE NULL::integer END) -> Seq Scan on public.int4_tbl Output: int4_tbl.f1 -> Seq Scan on public.tenk1 b @@ -3525,9 +3527,11 @@ SELECT * FROM onek t1, lateral (SELECT * FROM onek t2 WHERE t2.ten IN (values (t -> Hash Semi Join Hash Cond: (t2.ten = "*VALUES*".column1) -> Seq Scan on onek t2 + Bloom Filter 1: keys=(ten) -> Hash + Bloom Filter 1 -> Values Scan on "*VALUES*" -(7 rows) +(9 rows) -- VtA causes the whole expression to be evaluated as a constant EXPLAIN (COSTS OFF) @@ -3674,15 +3678,14 @@ WHERE id NOT IN ( Hash Cond: (not_null_tab.id = t2.id) -> Seq Scan on not_null_tab -> Hash - -> Merge Join - Merge Cond: (t1.id = t2.id) - -> Sort - Sort Key: t1.id - -> Seq Scan on not_null_tab t1 - -> Sort - Sort Key: t2.id + -> Hash Join + Hash Cond: (t1.id = t2.id) + -> Seq Scan on not_null_tab t1 + Bloom Filter 1: keys=(id) + -> Hash + Bloom Filter 1 -> Seq Scan on not_null_tab t2 -(12 rows) +(11 rows) -- ANTI JOIN: outer side is defined NOT NULL, inner side is forced nonnullable -- by qual clause @@ -3723,11 +3726,8 @@ WHERE id NOT IN ( ); QUERY PLAN ------------------------------------------------- - Merge Anti Join - Merge Cond: (not_null_tab.id = t1.id) - -> Sort - Sort Key: not_null_tab.id - -> Seq Scan on not_null_tab + Merge Right Anti Join + Merge Cond: (t1.id = not_null_tab.id) -> Nested Loop Left Join -> Merge Join Merge Cond: (t1.id = t2.id) @@ -3739,6 +3739,9 @@ WHERE id NOT IN ( -> Seq Scan on null_tab t2 -> Materialize -> Seq Scan on null_tab t3 + -> Sort + Sort Key: not_null_tab.id + -> Seq Scan on not_null_tab (16 rows) -- ANTI JOIN: outer side is defined NOT NULL and is not nulled by outer join, diff --git a/src/test/regress/expected/sysviews.out b/src/test/regress/expected/sysviews.out index 1e327c2afa4..3ad64918b46 100644 --- a/src/test/regress/expected/sysviews.out +++ b/src/test/regress/expected/sysviews.out @@ -164,6 +164,7 @@ select name, setting from pg_settings where name like 'enable%'; enable_groupagg | on enable_hashagg | on enable_hashjoin | on + enable_hashjoin_bloom | on enable_incremental_sort | on enable_indexonlyscan | on enable_indexscan | on @@ -181,7 +182,7 @@ select name, setting from pg_settings where name like 'enable%'; enable_seqscan | on enable_sort | on enable_tidscan | on -(26 rows) +(27 rows) -- There are always wait event descriptions for various types. InjectionPoint -- may be present or absent, depending on history since last postmaster start. diff --git a/src/test/regress/expected/updatable_views.out b/src/test/regress/expected/updatable_views.out index 9c6bb2219f9..f1a23f38a03 100644 --- a/src/test/regress/expected/updatable_views.out +++ b/src/test/regress/expected/updatable_views.out @@ -587,11 +587,13 @@ MERGE INTO rw_view1 t Hash Cond: (base_tbl.a = generate_series.generate_series) -> Bitmap Heap Scan on base_tbl Recheck Cond: (a > 0) + Bloom Filter 1: keys=(a) -> Bitmap Index Scan on base_tbl_pkey Index Cond: (a > 0) -> Hash + Bloom Filter 1 -> Function Scan on generate_series -(9 rows) +(11 rows) EXPLAIN (costs off) MERGE INTO rw_view1 t @@ -2777,12 +2779,14 @@ EXPLAIN (costs off) UPDATE rw_view1 SET a = a + 5; -> Hash Join Hash Cond: (b.a = r.a) -> Seq Scan on base_tbl b + Bloom Filter 1: keys=(a) -> Hash + Bloom Filter 1 -> Seq Scan on ref_tbl r SubPlan exists_1 -> Index Only Scan using ref_tbl_pkey on ref_tbl r_1 Index Cond: (a = b.a) -(9 rows) +(11 rows) DROP TABLE base_tbl, ref_tbl CASCADE; NOTICE: drop cascades to view rw_view1 @@ -3522,18 +3526,17 @@ EXPLAIN (COSTS OFF) UPDATE v2 SET a = 1; Update on t1 InitPlan exists_1 -> Result - -> Merge Join - Merge Cond: (t1.a = v1.a) - -> Sort - Sort Key: t1.a - -> Seq Scan on t1 - -> Sort - Sort Key: v1.a + -> Hash Join + Hash Cond: (t1.a = v1.a) + -> Seq Scan on t1 + Bloom Filter 1: keys=(a) + -> Hash + Bloom Filter 1 -> Subquery Scan on v1 -> Result One-Time Filter: (InitPlan exists_1).col1 -> Seq Scan on t1 t1_1 -(14 rows) +(13 rows) DROP VIEW v2; DROP VIEW v1; diff --git a/src/test/regress/expected/window.out b/src/test/regress/expected/window.out index c0bde1c5eec..d55bc161159 100644 --- a/src/test/regress/expected/window.out +++ b/src/test/regress/expected/window.out @@ -4360,15 +4360,14 @@ WHERE s.c = 1; Run Condition: (ntile(e2.salary) OVER w1 <= 1) -> Sort Sort Key: e1.depname - -> Merge Join - Merge Cond: (e1.empno = e2.empno) - -> Sort - Sort Key: e1.empno - -> Seq Scan on empsalary e1 - -> Sort - Sort Key: e2.empno + -> Hash Join + Hash Cond: (e1.empno = e2.empno) + -> Seq Scan on empsalary e1 + Bloom Filter 1: keys=(empno) + -> Hash + Bloom Filter 1 -> Seq Scan on empsalary e2 -(15 rows) +(14 rows) -- Ensure the run condition optimization is used in cases where the WindowFunc -- has a Var from another query level @@ -5685,10 +5684,12 @@ LIMIT 1; -> Hash Join Hash Cond: (t1.unique1 = t2.tenthous) -> Index Only Scan using tenk1_unique1 on tenk1 t1 + Bloom Filter 1: keys=(unique1) -> Hash + Bloom Filter 1 -> Seq Scan on tenk1 t2 Filter: (two = 1) -(9 rows) +(11 rows) -- Ensure we get a cheap total plan. This time use UNBOUNDED FOLLOWING, which -- needs to read all join rows to output the first WindowAgg row. diff --git a/src/test/regress/expected/with.out b/src/test/regress/expected/with.out index addb24896be..821ea3c5b0e 100644 --- a/src/test/regress/expected/with.out +++ b/src/test/regress/expected/with.out @@ -751,22 +751,20 @@ select * from search_graph order by seq; -> Recursive Union -> Seq Scan on pg_temp.graph0 g Output: g.f, g.t, g.label, ARRAY[ROW(g.f, g.t)] - -> Merge Join + -> Hash Join Output: g_1.f, g_1.t, g_1.label, array_cat(sg.seq, ARRAY[ROW(g_1.f, g_1.t)]) - Merge Cond: (g_1.f = sg.t) - -> Sort + Hash Cond: (g_1.f = sg.t) + -> Seq Scan on pg_temp.graph0 g_1 Output: g_1.f, g_1.t, g_1.label - Sort Key: g_1.f - -> Seq Scan on pg_temp.graph0 g_1 - Output: g_1.f, g_1.t, g_1.label - -> Sort + Bloom Filter 1: keys=(g_1.f) + -> Hash Output: sg.seq, sg.t - Sort Key: sg.t + Bloom Filter 1 -> WorkTable Scan on search_graph sg Output: sg.seq, sg.t -> CTE Scan on search_graph Output: search_graph.f, search_graph.t, search_graph.label, search_graph.seq -(22 rows) +(20 rows) with recursive search_graph(f, t, label) as ( select * from graph0 g @@ -824,22 +822,20 @@ select * from search_graph order by seq; -> Recursive Union -> Seq Scan on pg_temp.graph0 g Output: g.f, g.t, g.label, ROW('0'::bigint, g.f, g.t) - -> Merge Join + -> Hash Join Output: g_1.f, g_1.t, g_1.label, ROW(int8inc((sg.seq)."*DEPTH*"), g_1.f, g_1.t) - Merge Cond: (g_1.f = sg.t) - -> Sort + Hash Cond: (g_1.f = sg.t) + -> Seq Scan on pg_temp.graph0 g_1 Output: g_1.f, g_1.t, g_1.label - Sort Key: g_1.f - -> Seq Scan on pg_temp.graph0 g_1 - Output: g_1.f, g_1.t, g_1.label - -> Sort + Bloom Filter 1: keys=(g_1.f) + -> Hash Output: sg.seq, sg.t - Sort Key: sg.t + Bloom Filter 1 -> WorkTable Scan on search_graph sg Output: sg.seq, sg.t -> CTE Scan on search_graph Output: search_graph.f, search_graph.t, search_graph.label, search_graph.seq -(22 rows) +(20 rows) with recursive search_graph(f, t, label) as ( select * from graph0 g @@ -1095,20 +1091,20 @@ select * from search_graph; 5 | 1 | arc 5 -> 1 | f | {"(5,1)"} 1 | 2 | arc 1 -> 2 | f | {"(5,1)","(1,2)"} 1 | 3 | arc 1 -> 3 | f | {"(5,1)","(1,3)"} - 1 | 4 | arc 1 -> 4 | f | {"(5,1)","(1,4)"} 2 | 3 | arc 2 -> 3 | f | {"(1,2)","(2,3)"} + 1 | 4 | arc 1 -> 4 | f | {"(5,1)","(1,4)"} 4 | 5 | arc 4 -> 5 | f | {"(1,4)","(4,5)"} 5 | 1 | arc 5 -> 1 | f | {"(4,5)","(5,1)"} 1 | 2 | arc 1 -> 2 | f | {"(4,5)","(5,1)","(1,2)"} 1 | 3 | arc 1 -> 3 | f | {"(4,5)","(5,1)","(1,3)"} - 1 | 4 | arc 1 -> 4 | f | {"(4,5)","(5,1)","(1,4)"} 2 | 3 | arc 2 -> 3 | f | {"(5,1)","(1,2)","(2,3)"} + 1 | 4 | arc 1 -> 4 | f | {"(4,5)","(5,1)","(1,4)"} 4 | 5 | arc 4 -> 5 | f | {"(5,1)","(1,4)","(4,5)"} 5 | 1 | arc 5 -> 1 | f | {"(1,4)","(4,5)","(5,1)"} 1 | 2 | arc 1 -> 2 | f | {"(1,4)","(4,5)","(5,1)","(1,2)"} 1 | 3 | arc 1 -> 3 | f | {"(1,4)","(4,5)","(5,1)","(1,3)"} - 1 | 4 | arc 1 -> 4 | t | {"(1,4)","(4,5)","(5,1)","(1,4)"} 2 | 3 | arc 2 -> 3 | f | {"(4,5)","(5,1)","(1,2)","(2,3)"} + 1 | 4 | arc 1 -> 4 | t | {"(1,4)","(4,5)","(5,1)","(1,4)"} 4 | 5 | arc 4 -> 5 | t | {"(4,5)","(5,1)","(1,4)","(4,5)"} 5 | 1 | arc 5 -> 1 | t | {"(5,1)","(1,4)","(4,5)","(5,1)"} 2 | 3 | arc 2 -> 3 | f | {"(1,4)","(4,5)","(5,1)","(1,2)","(2,3)"} @@ -1133,20 +1129,20 @@ select * from search_graph; 5 | 1 | arc 5 -> 1 | f | {"(5,1)"} 1 | 2 | arc 1 -> 2 | f | {"(5,1)","(1,2)"} 1 | 3 | arc 1 -> 3 | f | {"(5,1)","(1,3)"} - 1 | 4 | arc 1 -> 4 | f | {"(5,1)","(1,4)"} 2 | 3 | arc 2 -> 3 | f | {"(1,2)","(2,3)"} + 1 | 4 | arc 1 -> 4 | f | {"(5,1)","(1,4)"} 4 | 5 | arc 4 -> 5 | f | {"(1,4)","(4,5)"} 5 | 1 | arc 5 -> 1 | f | {"(4,5)","(5,1)"} 1 | 2 | arc 1 -> 2 | f | {"(4,5)","(5,1)","(1,2)"} 1 | 3 | arc 1 -> 3 | f | {"(4,5)","(5,1)","(1,3)"} - 1 | 4 | arc 1 -> 4 | f | {"(4,5)","(5,1)","(1,4)"} 2 | 3 | arc 2 -> 3 | f | {"(5,1)","(1,2)","(2,3)"} + 1 | 4 | arc 1 -> 4 | f | {"(4,5)","(5,1)","(1,4)"} 4 | 5 | arc 4 -> 5 | f | {"(5,1)","(1,4)","(4,5)"} 5 | 1 | arc 5 -> 1 | f | {"(1,4)","(4,5)","(5,1)"} 1 | 2 | arc 1 -> 2 | f | {"(1,4)","(4,5)","(5,1)","(1,2)"} 1 | 3 | arc 1 -> 3 | f | {"(1,4)","(4,5)","(5,1)","(1,3)"} - 1 | 4 | arc 1 -> 4 | t | {"(1,4)","(4,5)","(5,1)","(1,4)"} 2 | 3 | arc 2 -> 3 | f | {"(4,5)","(5,1)","(1,2)","(2,3)"} + 1 | 4 | arc 1 -> 4 | t | {"(1,4)","(4,5)","(5,1)","(1,4)"} 4 | 5 | arc 4 -> 5 | t | {"(4,5)","(5,1)","(1,4)","(4,5)"} 5 | 1 | arc 5 -> 1 | t | {"(5,1)","(1,4)","(4,5)","(5,1)"} 2 | 3 | arc 2 -> 3 | f | {"(1,4)","(4,5)","(5,1)","(1,2)","(2,3)"} @@ -1208,21 +1204,19 @@ select * from search_graph; -> Recursive Union -> Seq Scan on pg_temp.graph g Output: g.f, g.t, g.label, false, ARRAY[ROW(g.f, g.t)] - -> Merge Join + -> Hash Join Output: g_1.f, g_1.t, g_1.label, CASE WHEN (ROW(g_1.f, g_1.t) = ANY (sg.path)) THEN true ELSE false END, array_cat(sg.path, ARRAY[ROW(g_1.f, g_1.t)]) - Merge Cond: (g_1.f = sg.t) - -> Sort + Hash Cond: (g_1.f = sg.t) + -> Seq Scan on pg_temp.graph g_1 Output: g_1.f, g_1.t, g_1.label - Sort Key: g_1.f - -> Seq Scan on pg_temp.graph g_1 - Output: g_1.f, g_1.t, g_1.label - -> Sort + Bloom Filter 1: keys=(g_1.f) + -> Hash Output: sg.path, sg.t - Sort Key: sg.t + Bloom Filter 1 -> WorkTable Scan on search_graph sg Output: sg.path, sg.t Filter: (NOT sg.is_cycle) -(20 rows) +(18 rows) with recursive search_graph(f, t, label) as ( select * from graph g @@ -1242,20 +1236,20 @@ select * from search_graph; 5 | 1 | arc 5 -> 1 | f | {"(5,1)"} 1 | 2 | arc 1 -> 2 | f | {"(5,1)","(1,2)"} 1 | 3 | arc 1 -> 3 | f | {"(5,1)","(1,3)"} - 1 | 4 | arc 1 -> 4 | f | {"(5,1)","(1,4)"} 2 | 3 | arc 2 -> 3 | f | {"(1,2)","(2,3)"} + 1 | 4 | arc 1 -> 4 | f | {"(5,1)","(1,4)"} 4 | 5 | arc 4 -> 5 | f | {"(1,4)","(4,5)"} 5 | 1 | arc 5 -> 1 | f | {"(4,5)","(5,1)"} 1 | 2 | arc 1 -> 2 | f | {"(4,5)","(5,1)","(1,2)"} 1 | 3 | arc 1 -> 3 | f | {"(4,5)","(5,1)","(1,3)"} - 1 | 4 | arc 1 -> 4 | f | {"(4,5)","(5,1)","(1,4)"} 2 | 3 | arc 2 -> 3 | f | {"(5,1)","(1,2)","(2,3)"} + 1 | 4 | arc 1 -> 4 | f | {"(4,5)","(5,1)","(1,4)"} 4 | 5 | arc 4 -> 5 | f | {"(5,1)","(1,4)","(4,5)"} 5 | 1 | arc 5 -> 1 | f | {"(1,4)","(4,5)","(5,1)"} 1 | 2 | arc 1 -> 2 | f | {"(1,4)","(4,5)","(5,1)","(1,2)"} 1 | 3 | arc 1 -> 3 | f | {"(1,4)","(4,5)","(5,1)","(1,3)"} - 1 | 4 | arc 1 -> 4 | t | {"(1,4)","(4,5)","(5,1)","(1,4)"} 2 | 3 | arc 2 -> 3 | f | {"(4,5)","(5,1)","(1,2)","(2,3)"} + 1 | 4 | arc 1 -> 4 | t | {"(1,4)","(4,5)","(5,1)","(1,4)"} 4 | 5 | arc 4 -> 5 | t | {"(4,5)","(5,1)","(1,4)","(4,5)"} 5 | 1 | arc 5 -> 1 | t | {"(5,1)","(1,4)","(4,5)","(5,1)"} 2 | 3 | arc 2 -> 3 | f | {"(1,4)","(4,5)","(5,1)","(1,2)","(2,3)"} @@ -1279,20 +1273,20 @@ select * from search_graph; 5 | 1 | arc 5 -> 1 | N | {"(5,1)"} 1 | 2 | arc 1 -> 2 | N | {"(5,1)","(1,2)"} 1 | 3 | arc 1 -> 3 | N | {"(5,1)","(1,3)"} - 1 | 4 | arc 1 -> 4 | N | {"(5,1)","(1,4)"} 2 | 3 | arc 2 -> 3 | N | {"(1,2)","(2,3)"} + 1 | 4 | arc 1 -> 4 | N | {"(5,1)","(1,4)"} 4 | 5 | arc 4 -> 5 | N | {"(1,4)","(4,5)"} 5 | 1 | arc 5 -> 1 | N | {"(4,5)","(5,1)"} 1 | 2 | arc 1 -> 2 | N | {"(4,5)","(5,1)","(1,2)"} 1 | 3 | arc 1 -> 3 | N | {"(4,5)","(5,1)","(1,3)"} - 1 | 4 | arc 1 -> 4 | N | {"(4,5)","(5,1)","(1,4)"} 2 | 3 | arc 2 -> 3 | N | {"(5,1)","(1,2)","(2,3)"} + 1 | 4 | arc 1 -> 4 | N | {"(4,5)","(5,1)","(1,4)"} 4 | 5 | arc 4 -> 5 | N | {"(5,1)","(1,4)","(4,5)"} 5 | 1 | arc 5 -> 1 | N | {"(1,4)","(4,5)","(5,1)"} 1 | 2 | arc 1 -> 2 | N | {"(1,4)","(4,5)","(5,1)","(1,2)"} 1 | 3 | arc 1 -> 3 | N | {"(1,4)","(4,5)","(5,1)","(1,3)"} - 1 | 4 | arc 1 -> 4 | Y | {"(1,4)","(4,5)","(5,1)","(1,4)"} 2 | 3 | arc 2 -> 3 | N | {"(4,5)","(5,1)","(1,2)","(2,3)"} + 1 | 4 | arc 1 -> 4 | Y | {"(1,4)","(4,5)","(5,1)","(1,4)"} 4 | 5 | arc 4 -> 5 | Y | {"(4,5)","(5,1)","(1,4)","(4,5)"} 5 | 1 | arc 5 -> 1 | Y | {"(5,1)","(1,4)","(4,5)","(5,1)"} 2 | 3 | arc 2 -> 3 | N | {"(1,4)","(4,5)","(5,1)","(1,2)","(2,3)"} @@ -1444,20 +1438,20 @@ select * from search_graph; 5 | 1 | arc 5 -> 1 | {"(5,1)"} | f | {"(5,1)"} 1 | 2 | arc 1 -> 2 | {"(5,1)","(1,2)"} | f | {"(5,1)","(1,2)"} 1 | 3 | arc 1 -> 3 | {"(5,1)","(1,3)"} | f | {"(5,1)","(1,3)"} - 1 | 4 | arc 1 -> 4 | {"(5,1)","(1,4)"} | f | {"(5,1)","(1,4)"} 2 | 3 | arc 2 -> 3 | {"(1,2)","(2,3)"} | f | {"(1,2)","(2,3)"} + 1 | 4 | arc 1 -> 4 | {"(5,1)","(1,4)"} | f | {"(5,1)","(1,4)"} 4 | 5 | arc 4 -> 5 | {"(1,4)","(4,5)"} | f | {"(1,4)","(4,5)"} 5 | 1 | arc 5 -> 1 | {"(4,5)","(5,1)"} | f | {"(4,5)","(5,1)"} 1 | 2 | arc 1 -> 2 | {"(4,5)","(5,1)","(1,2)"} | f | {"(4,5)","(5,1)","(1,2)"} 1 | 3 | arc 1 -> 3 | {"(4,5)","(5,1)","(1,3)"} | f | {"(4,5)","(5,1)","(1,3)"} - 1 | 4 | arc 1 -> 4 | {"(4,5)","(5,1)","(1,4)"} | f | {"(4,5)","(5,1)","(1,4)"} 2 | 3 | arc 2 -> 3 | {"(5,1)","(1,2)","(2,3)"} | f | {"(5,1)","(1,2)","(2,3)"} + 1 | 4 | arc 1 -> 4 | {"(4,5)","(5,1)","(1,4)"} | f | {"(4,5)","(5,1)","(1,4)"} 4 | 5 | arc 4 -> 5 | {"(5,1)","(1,4)","(4,5)"} | f | {"(5,1)","(1,4)","(4,5)"} 5 | 1 | arc 5 -> 1 | {"(1,4)","(4,5)","(5,1)"} | f | {"(1,4)","(4,5)","(5,1)"} 1 | 2 | arc 1 -> 2 | {"(1,4)","(4,5)","(5,1)","(1,2)"} | f | {"(1,4)","(4,5)","(5,1)","(1,2)"} 1 | 3 | arc 1 -> 3 | {"(1,4)","(4,5)","(5,1)","(1,3)"} | f | {"(1,4)","(4,5)","(5,1)","(1,3)"} - 1 | 4 | arc 1 -> 4 | {"(1,4)","(4,5)","(5,1)","(1,4)"} | t | {"(1,4)","(4,5)","(5,1)","(1,4)"} 2 | 3 | arc 2 -> 3 | {"(4,5)","(5,1)","(1,2)","(2,3)"} | f | {"(4,5)","(5,1)","(1,2)","(2,3)"} + 1 | 4 | arc 1 -> 4 | {"(1,4)","(4,5)","(5,1)","(1,4)"} | t | {"(1,4)","(4,5)","(5,1)","(1,4)"} 4 | 5 | arc 4 -> 5 | {"(4,5)","(5,1)","(1,4)","(4,5)"} | t | {"(4,5)","(5,1)","(1,4)","(4,5)"} 5 | 1 | arc 5 -> 1 | {"(5,1)","(1,4)","(4,5)","(5,1)"} | t | {"(5,1)","(1,4)","(4,5)","(5,1)"} 2 | 3 | arc 2 -> 3 | {"(1,4)","(4,5)","(5,1)","(1,2)","(2,3)"} | f | {"(1,4)","(4,5)","(5,1)","(1,2)","(2,3)"} @@ -1482,20 +1476,20 @@ select * from search_graph; 5 | 1 | arc 5 -> 1 | (0,5,1) | f | {"(5,1)"} 1 | 2 | arc 1 -> 2 | (1,1,2) | f | {"(5,1)","(1,2)"} 1 | 3 | arc 1 -> 3 | (1,1,3) | f | {"(5,1)","(1,3)"} - 1 | 4 | arc 1 -> 4 | (1,1,4) | f | {"(5,1)","(1,4)"} 2 | 3 | arc 2 -> 3 | (1,2,3) | f | {"(1,2)","(2,3)"} + 1 | 4 | arc 1 -> 4 | (1,1,4) | f | {"(5,1)","(1,4)"} 4 | 5 | arc 4 -> 5 | (1,4,5) | f | {"(1,4)","(4,5)"} 5 | 1 | arc 5 -> 1 | (1,5,1) | f | {"(4,5)","(5,1)"} 1 | 2 | arc 1 -> 2 | (2,1,2) | f | {"(4,5)","(5,1)","(1,2)"} 1 | 3 | arc 1 -> 3 | (2,1,3) | f | {"(4,5)","(5,1)","(1,3)"} - 1 | 4 | arc 1 -> 4 | (2,1,4) | f | {"(4,5)","(5,1)","(1,4)"} 2 | 3 | arc 2 -> 3 | (2,2,3) | f | {"(5,1)","(1,2)","(2,3)"} + 1 | 4 | arc 1 -> 4 | (2,1,4) | f | {"(4,5)","(5,1)","(1,4)"} 4 | 5 | arc 4 -> 5 | (2,4,5) | f | {"(5,1)","(1,4)","(4,5)"} 5 | 1 | arc 5 -> 1 | (2,5,1) | f | {"(1,4)","(4,5)","(5,1)"} 1 | 2 | arc 1 -> 2 | (3,1,2) | f | {"(1,4)","(4,5)","(5,1)","(1,2)"} 1 | 3 | arc 1 -> 3 | (3,1,3) | f | {"(1,4)","(4,5)","(5,1)","(1,3)"} - 1 | 4 | arc 1 -> 4 | (3,1,4) | t | {"(1,4)","(4,5)","(5,1)","(1,4)"} 2 | 3 | arc 2 -> 3 | (3,2,3) | f | {"(4,5)","(5,1)","(1,2)","(2,3)"} + 1 | 4 | arc 1 -> 4 | (3,1,4) | t | {"(1,4)","(4,5)","(5,1)","(1,4)"} 4 | 5 | arc 4 -> 5 | (3,4,5) | t | {"(4,5)","(5,1)","(1,4)","(4,5)"} 5 | 1 | arc 5 -> 1 | (3,5,1) | t | {"(5,1)","(1,4)","(4,5)","(5,1)"} 2 | 3 | arc 2 -> 3 | (4,2,3) | f | {"(1,4)","(4,5)","(5,1)","(1,2)","(2,3)"} @@ -1675,20 +1669,20 @@ select * from v_cycle1; 5 | 1 | arc 5 -> 1 1 | 2 | arc 1 -> 2 1 | 3 | arc 1 -> 3 - 1 | 4 | arc 1 -> 4 2 | 3 | arc 2 -> 3 + 1 | 4 | arc 1 -> 4 4 | 5 | arc 4 -> 5 5 | 1 | arc 5 -> 1 1 | 2 | arc 1 -> 2 1 | 3 | arc 1 -> 3 - 1 | 4 | arc 1 -> 4 2 | 3 | arc 2 -> 3 + 1 | 4 | arc 1 -> 4 4 | 5 | arc 4 -> 5 5 | 1 | arc 5 -> 1 1 | 2 | arc 1 -> 2 1 | 3 | arc 1 -> 3 - 1 | 4 | arc 1 -> 4 2 | 3 | arc 2 -> 3 + 1 | 4 | arc 1 -> 4 4 | 5 | arc 4 -> 5 5 | 1 | arc 5 -> 1 2 | 3 | arc 2 -> 3 @@ -1705,20 +1699,20 @@ select * from v_cycle2; 5 | 1 | arc 5 -> 1 1 | 2 | arc 1 -> 2 1 | 3 | arc 1 -> 3 - 1 | 4 | arc 1 -> 4 2 | 3 | arc 2 -> 3 + 1 | 4 | arc 1 -> 4 4 | 5 | arc 4 -> 5 5 | 1 | arc 5 -> 1 1 | 2 | arc 1 -> 2 1 | 3 | arc 1 -> 3 - 1 | 4 | arc 1 -> 4 2 | 3 | arc 2 -> 3 + 1 | 4 | arc 1 -> 4 4 | 5 | arc 4 -> 5 5 | 1 | arc 5 -> 1 1 | 2 | arc 1 -> 2 1 | 3 | arc 1 -> 3 - 1 | 4 | arc 1 -> 4 2 | 3 | arc 2 -> 3 + 1 | 4 | arc 1 -> 4 4 | 5 | arc 4 -> 5 5 | 1 | arc 5 -> 1 2 | 3 | arc 2 -> 3 diff --git a/src/test/regress/sql/rowsecurity.sql b/src/test/regress/sql/rowsecurity.sql index 6b3566271df..390e79b04c3 100644 --- a/src/test/regress/sql/rowsecurity.sql +++ b/src/test/regress/sql/rowsecurity.sql @@ -2,6 +2,9 @@ -- Test of Row-level security feature -- +-- disable bloom filter pushdown, to not interfere with calls to functions +SET enable_hashjoin_bloom = off; + -- Clean up in case a prior regression run failed -- Suppress NOTICE messages when users/groups don't exist diff --git a/src/test/regress/sql/select_views.sql b/src/test/regress/sql/select_views.sql index e742f136990..09f96b0b1ae 100644 --- a/src/test/regress/sql/select_views.sql +++ b/src/test/regress/sql/select_views.sql @@ -3,6 +3,9 @@ -- test the views defined in CREATE_VIEWS -- +-- disable bloom filter pushdown, to not interfere with calls to functions +SET enable_hashjoin_bloom = off; + SELECT * FROM street; SELECT name, #thepath FROM iexit ORDER BY name COLLATE "C", 2; diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 56c1f997f88..4acac4e84a1 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -316,6 +316,7 @@ BlocksReadStreamData BlocktableEntry BloomBuildState BloomFilter +BloomFilterState BloomMetaPageData BloomOpaque BloomOptions @@ -802,6 +803,7 @@ ExpandedObjectMethods ExpandedRange ExpandedRecordFieldInfo ExpandedRecordHeader +ExpectedFilter ExplainDirectModify_function ExplainExtensionOption ExplainForeignModify_function -- 2.55.0