From 9d200f3d0a19f14f335eb48034ad720b5e944ede Mon Sep 17 00:00:00 2001 From: Alexandra Wang Date: Mon, 3 Aug 2026 15:59:52 -0700 Subject: [PATCH v10 4/4] Add join MCV statistics for selectivity estimation Extends the extended statistics framework to support MCV lists that span an equijoin between tables. This lets the planner use per-value join frequencies instead of uniform estimates (1/ndistinct or 1/ref_tuples) when filter predicates are present on the joined tables. Syntax: CREATE STATISTICS s (mcv) ON d.name FROM fact f JOIN dim d ON (f.dim_id = d.id); A join statistics object describes the rows of SELECT ... FROM , which is parsed as for a SELECT, so USING, NATURAL and parenthesized joins are accepted. The inner joins are flattened into a set of join conditions; each must be an equality between columns of two tables, and together they must connect all the tables. Joins of more than two tables and several conditions between two tables are valid definitions, but are not built yet (see below). The pg_statistic_ext catalog gains two nullable columns, used only by join statistics: stxjoinrels, the joined tables with the anchor first, and stxjoinconds, the join conditions. Each column in stxexprs identifies its table by its varno. We call the first table in the FROM clause the "anchor" table (the fact table in the above example). ANALYZE of the anchor table samples it and looks up the matching rows of the other table through an index on its join column, once for each distinct join value in the sample, following the index-based join sampling of "Cardinality Estimation Done Right: Index-Based Join Sampling" (Leis et al., CIDR 2017). A uniform sample of the joined rows is built in a single pass, and the result is a standard MCV list. Currently only a join of two tables on a single equality condition is built; other shapes are skipped with a warning. Join statistics estimate how often the filters on the joined tables hold among the rows of the join: an MCV frequency is the fraction of the joined rows having those values. The size of the join itself still comes from the planner's estimate of the join clause, from eqjoinsel or a matching foreign key. That estimate is multiplied by the ratio of the measured frequency to the filters' selectivity in their own tables, so filters unrelated to the join leave it unchanged. A statistics object is applied once, at the join step whose two sides together first contain its tables, and only when the query joins those tables on exactly the condition it was built on; a join of the same tables on more conditions is estimated as without it. Parameterized scans get the same correction through clause_selectivity_ext(). Semi and anti joins are not corrected, as the statistics do not record which rows have a match. Filters are matched against the MCV items by the single-table MCV code, so the same filter forms are covered. ANALYZE needs an index on the probed table's join column; CREATE STATISTICS warns when there is none, and ANALYZE skips the statistics object, keeping its data, until one exists. The object does not depend on the index: the statistics object belongs with the anchor table, while the index belongs with the probed table, whose owner can drop or replace it. A later patch can also let ANALYZE sample by scanning the probed table instead. An unnamed join statistics object qualifies its column names with their table names, so an object on (t1.b, t1.c, t2.e, t2.f) is named t1_b_c_t2_e_f_stat. Because join statistics read more than one table, creating one requires ownership of the anchor table and SELECT on the other table(s). ANALYZE rechecks that SELECT for the statistics object's owner, and skips the refresh with a warning if it has been revoked. pg_stats_ext applies its visibility check to every joined table. pg_restore_extended_stats() needs privileges only on the anchor table, as it writes only values supplied by the caller. Reference: https://www.cidrdb.org/cidr2017/papers/p9-leis-cidr17.pdf Reviewed-by: Tomas Vondra Reviewed-by: Tom Lane Reviewed-by: Corey Huinker Reviewed-by: Chengpeng Yan Reviewed-by: Ilia Evdokimov Discussion: https://www.postgresql.org/message-id/flat/CADkLM%3DcUwMftPLFq0iD6-qKRyNiRM2HZGYVp6%3D0noxA8GfuEtA%40mail.gmail.com --- doc/src/sgml/catalogs.sgml | 29 +- doc/src/sgml/perform.sgml | 17 + doc/src/sgml/ref/create_statistics.sgml | 123 +- doc/src/sgml/ref/create_table.sgml | 4 +- src/backend/catalog/system_views.sql | 12 + src/backend/commands/statscmds.c | 683 +++- src/backend/commands/tablecmds.c | 36 + src/backend/optimizer/path/clausesel.c | 9 + src/backend/optimizer/path/costsize.c | 21 +- src/backend/optimizer/path/equivclass.c | 79 + src/backend/optimizer/util/joininfo.c | 50 + src/backend/optimizer/util/plancat.c | 44 +- src/backend/parser/gram.y | 7 + src/backend/parser/parse_utilcmd.c | 363 +- src/backend/statistics/Makefile | 1 + src/backend/statistics/extended_stats.c | 323 +- src/backend/statistics/extended_stats_funcs.c | 37 +- src/backend/statistics/join_mcv.c | 1469 ++++++++ src/backend/statistics/mcv.c | 7 +- src/backend/statistics/meson.build | 1 + src/backend/tcop/utility.c | 69 +- src/backend/utils/adt/ruleutils.c | 220 +- src/bin/psql/describe.c | 57 +- src/include/catalog/pg_proc.dat | 3 +- src/include/catalog/pg_statistic_ext.h | 9 + src/include/nodes/parsenodes.h | 6 + src/include/nodes/pathnodes.h | 12 + src/include/optimizer/joininfo.h | 2 + src/include/optimizer/paths.h | 3 + .../statistics/extended_stats_internal.h | 26 +- src/include/statistics/statistics.h | 14 + .../isolation/expected/join-stats-analyze.out | 12 + src/test/isolation/isolation_schedule | 1 + .../isolation/specs/join-stats-analyze.spec | 41 + src/test/regress/expected/oidjoins.out | 1 + src/test/regress/expected/rules.out | 10 +- src/test/regress/expected/stats_ext.out | 27 +- .../regress/expected/stats_ext_crossrel.out | 3058 +++++++++++++++++ src/test/regress/expected/stats_import.out | 154 +- src/test/regress/parallel_schedule | 3 + src/test/regress/sql/stats_ext.sql | 7 +- src/test/regress/sql/stats_ext_crossrel.sql | 2109 ++++++++++++ src/test/regress/sql/stats_import.sql | 118 + src/tools/pgindent/typedefs.list | 6 + 44 files changed, 8999 insertions(+), 284 deletions(-) create mode 100644 src/backend/statistics/join_mcv.c create mode 100644 src/test/isolation/expected/join-stats-analyze.out create mode 100644 src/test/isolation/specs/join-stats-analyze.spec create mode 100644 src/test/regress/expected/stats_ext_crossrel.out create mode 100644 src/test/regress/sql/stats_ext_crossrel.sql diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index 3703858b2a7..4aaba8d77be 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -7803,7 +7803,8 @@ SCRAM-SHA-256$<iteration count>:&l (references pg_class.oid) - Table containing the columns described by this object + Table containing the columns described by this object, or for join + statistics the first table of the join, which owns the object @@ -7881,6 +7882,32 @@ SCRAM-SHA-256$<iteration count>:&l + + + stxjoinrels oidvector + (references pg_class.oid) + + + For join statistics, the tables of the join, starting with + stxrelid. A column reference in + stxexprs or + stxjoinconds identifies its table by its + position in this list. Null for single-table statistics. + + + + + + stxjoinconds pg_node_tree + + + For join statistics, expression trees (in + nodeToString() representation) for the join + conditions, one element per equality between two columns. Null for + single-table statistics. + + + diff --git a/doc/src/sgml/perform.sgml b/doc/src/sgml/perform.sgml index b4e695a263d..62fa59304b0 100644 --- a/doc/src/sgml/perform.sgml +++ b/doc/src/sgml/perform.sgml @@ -1459,6 +1459,12 @@ WHERE tablename = 'road'; such information. + + Correlation can also span a join, which per-table statistics cannot + describe either. For this, statistics can be collected over the rows of + a join. + + Because the number of possible column combinations is very large, it's impractical to compute multivariate statistics automatically. @@ -1769,6 +1775,17 @@ SELECT m.* FROM pg_statistic_ext join pg_statistic_ext_data on (oid = stxoid), plans. Otherwise, the ANALYZE and planning cycles are just wasted. + + + An MCV list can also be collected over the rows of a + join, recording how often combinations of values from the joined tables + occur together. When a query joins those tables and restricts the listed + columns, the planner uses the join MCV list instead of + assuming the restrictions are independent of the join. Such lists are + created with the FROM ... JOIN form of + CREATE STATISTICS, + which describes what is currently supported. + diff --git a/doc/src/sgml/ref/create_statistics.sgml b/doc/src/sgml/ref/create_statistics.sgml index 5cc1d51b4b3..91a74ea7466 100644 --- a/doc/src/sgml/ref/create_statistics.sgml +++ b/doc/src/sgml/ref/create_statistics.sgml @@ -29,6 +29,22 @@ CREATE STATISTICS [ [ IF NOT EXISTS ] statistics_ [ ( statistics_kind [, ... ] ) ] ON { column_name | ( expression ) }, { column_name | ( expression ) } [, ...] FROM table_name + +CREATE STATISTICS [ [ IF NOT EXISTS ] statistics_name ] + [ ( statistics_kind [, ... ] ) ] + ON { table_name | alias }.column_name [, ...] + FROM join + +where join can be one of: + + join_item [ INNER ] JOIN join_item { ON join_condition | USING ( join_column [, ...] ) } + join_item NATURAL [ INNER ] JOIN join_item + +and join_item can be one of: + + table_name [ [ AS ] alias ] + join + ( join ) @@ -57,6 +73,15 @@ CREATE STATISTICS [ [ IF NOT EXISTS ] statistics_ any expressions and virtual generated columns included in the list. + + The third form collects join statistics: an + MCV (most-common-values) list over the rows of a join, + recording how often combinations of values from the joined tables occur + together. This captures correlations across the join that per-table + statistics cannot describe. Each column must be qualified with its table + name, or with the table's alias if it has one. + + If a schema name is given (for example, CREATE STATISTICS myschema.mystat ...) then the statistics object is created in the @@ -160,6 +185,12 @@ CREATE STATISTICS [ [ IF NOT EXISTS ] statistics_ linkend="sql-analyze"/> for an explanation of the handling of inheritance and partitions. + + For join statistics, the FROM clause joins two or + more tables as in . The first table listed + owns the statistics object, and ANALYZE of that + table builds the statistics. + @@ -175,6 +206,13 @@ CREATE STATISTICS [ [ IF NOT EXISTS ] statistics_ object is independent of the underlying table(s). + + Join statistics also require SELECT privilege on the + other tables. ANALYZE rechecks it for the statistics + object's owner, and skips the object, keeping any existing data, if it has + been revoked. + + Expression statistics are per-expression and are similar to creating an index on the expression, except that they avoid the overhead of index @@ -185,9 +223,38 @@ CREATE STATISTICS [ [ IF NOT EXISTS ] statistics_ - Extended statistics are not currently used by the planner for selectivity - estimations made for table joins. This limitation will likely be removed - in a future version of PostgreSQL. + The FROM clause of join statistics must be an inner + join, each join condition must be an equality between columns of two + tables, and the conditions together must connect all the tables. + Partitioned tables and tables with inheritance children are not + supported, and mcv is the only statistics kind. + + + + ANALYZE currently builds join statistics only for a join + of two tables on a single condition, and skips other definitions with a + warning. It finds the joining rows of the second table through an index, + so that table needs a non-partial index whose first column is the join + column. CREATE STATISTICS warns when there is none, and + ANALYZE skips the statistics object, keeping any + existing data, until one exists. The statistics object does not depend on + a particular index, so an index can be dropped or replaced freely. + + + + Extended statistics on a single table are not used for estimating joins. + Join statistics correct the planner's estimate of a join for the query's + restrictions on their columns. The estimate of the join itself still + comes from the per-column statistics of the join columns, or from a + foreign key constraint between them. The MCV list + records how often each combination of values occurs among the joined rows, + so the planner can tell whether the rows satisfying the restrictions join + more or less often than other rows of their tables. The statistics are + used for a join of their tables on exactly the condition they were built + on, outer joins included, but not for a join on further conditions, nor + for semi and anti joins such as EXISTS and + NOT EXISTS subqueries, whose estimate depends on which + rows have a match, which the statistics do not record. @@ -326,6 +393,56 @@ EXPLAIN ANALYZE SELECT date_trunc('month', a), date_trunc('day', a) more accurate estimates. + + Create a dimension table dims and a fact table + facts whose dim_id + values are heavily skewed toward one dimension row, then build a join MCV + list over the join of the two tables: + + +CREATE TABLE dims ( + id int PRIMARY KEY, + kind text +); + +CREATE TABLE facts ( + id int, + dim_id int +); + +-- 100 dimension rows, only one of which has kind 'hot': +INSERT INTO dims SELECT i, CASE WHEN i = 1 THEN 'hot' ELSE 'cold' END + FROM generate_series(1, 100) s(i); + +-- 100000 fact rows, about 90% referencing the single 'hot' dimension row: +INSERT INTO facts SELECT i, CASE WHEN i % 10 = 0 THEN 1 + (i % 100) ELSE 1 END + FROM generate_series(1, 100000) s(i); + +ANALYZE facts, dims; + +-- the join row count is drastically underestimated, because the planner +-- assumes fact rows are spread evenly over the dimension: +EXPLAIN ANALYZE SELECT * FROM facts f JOIN dims d ON (f.dim_id = d.id) + WHERE d.kind = 'hot'; + +-- build a join MCV list (dims.id is the primary key, and so is indexed): +CREATE STATISTICS s4 (mcv) ON d.kind + FROM facts f JOIN dims d ON (f.dim_id = d.id); + +ANALYZE facts; + +-- now the join row count estimate reflects the skew: +EXPLAIN ANALYZE SELECT * FROM facts f JOIN dims d ON (f.dim_id = d.id) + WHERE d.kind = 'hot'; + + + Without join statistics, the planner assumes the 'hot' + dimension row is joined by an average share of the fact rows, 1000 rows + here. The join MCV list records that about 90% of the + joined rows have kind = 'hot', and the estimate becomes + close to the actual 91000 rows. + + diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index fef24d8f3a2..e34c664adfd 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -778,7 +778,9 @@ WITH ( MODULUS numeric_literal, REM INCLUDING STATISTICS - Extended statistics are copied to the new table. + Extended statistics are copied to the new table. Join statistics + are not, as they describe a join of the table with other tables + rather than the table itself. diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index f1e16db7340..0213855833f 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -306,6 +306,12 @@ CREATE VIEW pg_stats_ext WITH (security_barrier) AS FROM pg_mcv_list_items(sd.stxdmcv) ) m ON sd.stxdmcv IS NOT NULL WHERE pg_has_role(c.relowner, 'USAGE') + AND (s.stxjoinrels IS NULL OR NOT EXISTS ( + SELECT 1 FROM unnest(s.stxjoinrels) AS jr(oid) + JOIN pg_class jc ON jc.oid = jr.oid + WHERE NOT pg_has_role(jc.relowner, 'USAGE') + OR (jc.relrowsecurity AND row_security_active(jc.oid)) + )) AND (c.relrowsecurity = false OR NOT row_security_active(c.oid)); CREATE VIEW pg_stats_ext_exprs WITH (security_barrier) AS @@ -400,6 +406,12 @@ CREATE VIEW pg_stats_ext_exprs WITH (security_barrier) AS unnest(sd.stxdexpr)::pg_statistic AS a ) stat ON (stat.expr IS NOT NULL) WHERE pg_has_role(c.relowner, 'USAGE') + AND (s.stxjoinrels IS NULL OR NOT EXISTS ( + SELECT 1 FROM unnest(s.stxjoinrels) AS jr(oid) + JOIN pg_class jc ON jc.oid = jr.oid + WHERE NOT pg_has_role(jc.relowner, 'USAGE') + OR (jc.relrowsecurity AND row_security_active(jc.oid)) + )) AND (c.relrowsecurity = false OR NOT row_security_active(c.oid)); -- unprivileged users may read pg_statistic_ext but not pg_statistic_ext_data diff --git a/src/backend/commands/statscmds.c b/src/backend/commands/statscmds.c index ea758a70a76..431f9ca8c8a 100644 --- a/src/backend/commands/statscmds.c +++ b/src/backend/commands/statscmds.c @@ -14,6 +14,7 @@ */ #include "postgres.h" +#include "access/genam.h" #include "access/htup_details.h" #include "access/relation.h" #include "access/table.h" @@ -23,6 +24,7 @@ #include "catalog/indexing.h" #include "catalog/namespace.h" #include "catalog/objectaccess.h" +#include "catalog/pg_inherits.h" #include "catalog/pg_namespace.h" #include "catalog/pg_statistic_ext.h" #include "catalog/pg_statistic_ext_data.h" @@ -32,6 +34,7 @@ #include "nodes/makefuncs.h" #include "nodes/nodeFuncs.h" #include "optimizer/optimizer.h" +#include "statistics/extended_stats_internal.h" #include "statistics/statistics.h" #include "utils/acl.h" #include "utils/builtins.h" @@ -44,7 +47,16 @@ static char *ChooseExtendedStatisticName(const char *name1, const char *name2, const char *label, Oid namespaceid); -static char *ChooseExtendedStatisticNameAddition(List *exprs, Oid relid); +static char *ChooseExtendedStatisticNameAddition(List *exprs, Oid relid, + List *joinrels); +static void check_join_stats_rel(Relation rel); +static List *open_join_stats_rels(CreateStatsStmt *stmt, Oid stxowner, + bool check_rights, Relation *anchor_rel); +static void check_join_stats_indexes(CreateStatsStmt *stmt, List *other_rels); +static void check_join_stats_columns(CreateStatsStmt *stmt, List *stxexprs); +static void record_join_stats_dependencies(const ObjectAddress *myself, + CreateStatsStmt *stmt, + List *stxexprs); /* * CREATE STATISTICS @@ -52,8 +64,7 @@ static char *ChooseExtendedStatisticNameAddition(List *exprs, Oid relid); * relids is a list of OIDs of relations specified in the FROM clause, on which * the statistics object is defined. We identify the target by the passed-in * OID rather than re-resolving stmt->relations by name, so that we operate - * on exactly the relation the caller looked up. Only a single relation is - * supported for now. + * on exactly the relation the caller looked up. */ ObjectAddress CreateStatistics(List *relids, CreateStatsStmt *stmt, bool check_rights) @@ -87,136 +98,97 @@ CreateStatistics(List *relids, CreateStatsStmt *stmt, bool check_rights) bool requested_type = false; ListCell *cell; ListCell *cell2; + ListCell *lc; + Node *rln; + bool isjoin = false; + List *other_rels = NIL; Assert(IsA(stmt, CreateStatsStmt)); /* - * Currently, we only allow the FROM clause to be a single simple table, - * but later we'll probably allow multiple tables and JOIN syntax. The - * grammar and the loop below are already prepared for that, but examining - * the FROM clause is the caller's job, so all we do here is assert that - * the caller rejected what we can't support. + * Examine the FROM clause. We support either: 1. Single RangeVar for + * single-table statistics 2. JoinExpr for join statistics */ - Assert(list_length(relids) == 1); + Assert(list_length(stmt->relations) == 1); + + rln = (Node *) linitial(stmt->relations); - foreach(cell, relids) + if (IsA(rln, RangeVar)) { - relid = lfirst_oid(cell); + Assert(list_length(relids) == 1); - /* - * CREATE STATISTICS will influence future execution plans but does - * not interfere with currently executing plans. So it should be - * enough to take only ShareUpdateExclusiveLock on relation, - * conflicting with ANALYZE and other DDL that sets statistical - * information, but not with normal queries. - */ - rel = relation_open(relid, ShareUpdateExclusiveLock); + foreach(cell, relids) + { + relid = lfirst_oid(cell); - /* Restrict to allowed relation types */ - if (rel->rd_rel->relkind != RELKIND_RELATION && - rel->rd_rel->relkind != RELKIND_MATVIEW && - rel->rd_rel->relkind != RELKIND_FOREIGN_TABLE && - rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE) - ereport(ERROR, - (errcode(ERRCODE_WRONG_OBJECT_TYPE), - errmsg("cannot define statistics for relation \"%s\"", - RelationGetRelationName(rel)), - errdetail_relkind_not_supported(rel->rd_rel->relkind))); + /* + * CREATE STATISTICS will influence future execution plans but + * does not interfere with currently executing plans. So it + * should be enough to take only ShareUpdateExclusiveLock on + * relation, conflicting with ANALYZE and other DDL that sets + * statistical information, but not with normal queries. + */ + rel = relation_open(relid, ShareUpdateExclusiveLock); - /* - * You must own the relation to create stats on it. Skip check if - * caller doesn't want it. - */ - if (check_rights && - !object_ownercheck(RelationRelationId, RelationGetRelid(rel), stxowner)) - aclcheck_error(ACLCHECK_NOT_OWNER, get_relkind_objtype(rel->rd_rel->relkind), - RelationGetRelationName(rel)); + /* Restrict to allowed relation types */ + if (rel->rd_rel->relkind != RELKIND_RELATION && + rel->rd_rel->relkind != RELKIND_MATVIEW && + rel->rd_rel->relkind != RELKIND_FOREIGN_TABLE && + rel->rd_rel->relkind != RELKIND_PARTITIONED_TABLE) + ereport(ERROR, + (errcode(ERRCODE_WRONG_OBJECT_TYPE), + errmsg("cannot define statistics for relation \"%s\"", + RelationGetRelationName(rel)), + errdetail_relkind_not_supported(rel->rd_rel->relkind))); - /* - * Conflict log tables are system-managed tables used internally for - * logical replication conflict logging. Unlike user tables, they are - * not expected to have complex query usage, so to keep things simple, - * user-defined extended statistics are not required or supported at - * present. - */ - if (IsConflictLogTableClass(rel->rd_rel)) - ereport(ERROR, - (errcode(ERRCODE_WRONG_OBJECT_TYPE), - errmsg("cannot create statistics on conflict log table \"%s\"", - RelationGetRelationName(rel)), - errdetail("Conflict log tables are system-managed tables for logical replication conflicts."))); + /* + * You must own the relation to create stats on it. Skip check if + * caller doesn't want it. + */ + if (check_rights && + !object_ownercheck(RelationRelationId, RelationGetRelid(rel), stxowner)) + aclcheck_error(ACLCHECK_NOT_OWNER, get_relkind_objtype(rel->rd_rel->relkind), + RelationGetRelationName(rel)); - /* Creating statistics on system catalogs is not allowed */ - if (!allowSystemTableMods && IsSystemRelation(rel)) - ereport(ERROR, - (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE), - errmsg("permission denied: \"%s\" is a system catalog", - RelationGetRelationName(rel)))); - } + /* + * Conflict log tables are system-managed tables used internally + * for logical replication conflict logging. Unlike user tables, + * they are not expected to have complex query usage, so to keep + * things simple, user-defined extended statistics are not + * required or supported at present. + */ + if (IsConflictLogTableClass(rel->rd_rel)) + ereport(ERROR, + (errcode(ERRCODE_WRONG_OBJECT_TYPE), + errmsg("cannot create statistics on conflict log table \"%s\"", + RelationGetRelationName(rel)), + errdetail("Conflict log tables are system-managed tables for logical replication conflicts."))); - Assert(rel); + /* Creating statistics on system catalogs is not allowed */ + if (!allowSystemTableMods && IsSystemRelation(rel)) + ereport(ERROR, + (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE), + errmsg("permission denied: \"%s\" is a system catalog", + RelationGetRelationName(rel)))); + } - /* - * If the node has a name, split it up and determine creation namespace. - * If not, put the object in the same namespace as the relation, and cons - * up a name for it. (This can happen either via "CREATE STATISTICS ..." - * or via "CREATE TABLE ... (LIKE)".) - */ - if (stmt->defnames) - namespaceId = QualifiedNameGetCreationNamespace(stmt->defnames, - &namestr); - else - { - namespaceId = RelationGetNamespace(rel); - namestr = ChooseExtendedStatisticName(RelationGetRelationName(rel), - ChooseExtendedStatisticNameAddition(stmt->exprs, - relid), - "stat", - namespaceId); + relid = RelationGetRelid(rel); } - namestrcpy(&stxname, namestr); - - /* - * Check we have creation rights in target namespace. Skip check if - * caller doesn't want it. - */ - if (check_rights) + else if (IsA(rln, JoinExpr)) { - AclResult aclresult; - - aclresult = object_aclcheck(NamespaceRelationId, namespaceId, - GetUserId(), ACL_CREATE); - if (aclresult != ACLCHECK_OK) - aclcheck_error(aclresult, OBJECT_SCHEMA, - get_namespace_name(namespaceId)); + Assert(stmt->transformed); + isjoin = true; + relid = stmt->stxrelid; } - - /* - * Deal with the possibility that the statistics object already exists. - */ - if (SearchSysCacheExists2(STATEXTNAMENSP, - CStringGetDatum(namestr), - ObjectIdGetDatum(namespaceId))) + else { - if (stmt->if_not_exists) - { - /* - * Since stats objects aren't members of extensions (see comments - * below), no need for checkMembershipInCurrentExtension here. - */ - ereport(NOTICE, - (errcode(ERRCODE_DUPLICATE_OBJECT), - errmsg("statistics object \"%s\" already exists, skipping", - namestr))); - relation_close(rel, NoLock); - return InvalidObjectAddress; - } - ereport(ERROR, - (errcode(ERRCODE_DUPLICATE_OBJECT), - errmsg("statistics object \"%s\" already exists", namestr))); + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("only a single relation or JOIN is allowed in CREATE STATISTICS"))); } + Assert(rel || isjoin); + /* * Make sure no more than STATS_MAX_DIMENSIONS columns are used. There * might be duplicates and so on, but we'll deal with those later. @@ -253,6 +225,12 @@ CreateStatistics(List *relids, CreateStatsStmt *stmt, bool check_rights) Form_pg_attribute attForm; TypeCacheEntry *type; + /* Join stats require table-qualified column names */ + if (isjoin) + ereport(ERROR, + (errcode(ERRCODE_INVALID_COLUMN_REFERENCE), + errmsg("join statistics require table-qualified column names"))); + attname = selem->name; atttuple = SearchSysCacheAttName(relid, attname); @@ -308,18 +286,28 @@ CreateStatistics(List *relids, CreateStatsStmt *stmt, bool check_rights) /* * Disallow data types without a less-than operator in - * multivariate statistics. + * multivariate statistics. Join stats always need this because + * the MCV builder requires a sort operator. */ - if (numcols > 1) + if (isjoin || numcols > 1) { + Oid col_relid; + type = lookup_type_cache(var->vartype, TYPECACHE_LT_OPR); if (type->lt_opr == InvalidOid) + { + if (isjoin) + col_relid = list_nth_oid(stmt->stxjoinrels, + var->varno - 1); + else + col_relid = relid; ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot create multivariate statistics on column \"%s\"", - get_attname(relid, var->varattno, false)), + get_attname(col_relid, var->varattno, false)), errdetail("The type %s has no default btree operator class.", format_type_be(var->vartype)))); + } } stxexprs = lappend(stxexprs, (Node *) var); @@ -330,15 +318,21 @@ CreateStatistics(List *relids, CreateStatsStmt *stmt, bool check_rights) Node *expr = selem->expr; Oid atttype; TypeCacheEntry *type; - Bitmapset *expr_attrs = NULL; + Bitmapset *attnums = NULL; int k; Assert(expr != NULL); - pull_varattnos(expr, 1, &expr_attrs); + /* Join stats only support simple column references */ + if (isjoin) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("expressions are not supported in join statistics"))); + + pull_varattnos(expr, 1, &attnums); k = -1; - while ((k = bms_next_member(expr_attrs, k)) >= 0) + while ((k = bms_next_member(attnums, k)) >= 0) { AttrNumber attnum = k + FirstLowInvalidHeapAttributeNumber; @@ -377,12 +371,100 @@ CreateStatistics(List *relids, CreateStatsStmt *stmt, bool check_rights) } } + /* + * For join statistics, open and validate all participating relations. The + * join info (stxrelid, stxjoinrels, stxjoinconds) was already resolved + * and validated by transformStatsStmt(). + */ + if (isjoin) + { + other_rels = open_join_stats_rels(stmt, stxowner, check_rights, &rel); + check_join_stats_indexes(stmt, other_rels); + check_join_stats_columns(stmt, stxexprs); + } + + /* + * Now determine namespace and name. Use rel (anchor table for joins). + */ + if (stmt->defnames) + namespaceId = QualifiedNameGetCreationNamespace(stmt->defnames, + &namestr); + else + { + namespaceId = RelationGetNamespace(rel); + + /* + * Join stats have no single relation to use as a name prefix, so the + * qualified column list is the whole name. + */ + if (isjoin) + namestr = ChooseExtendedStatisticName(ChooseExtendedStatisticNameAddition(stmt->exprs, + relid, + stmt->stxjoinrels), + NULL, + "stat", + namespaceId); + else + namestr = ChooseExtendedStatisticName(RelationGetRelationName(rel), + ChooseExtendedStatisticNameAddition(stmt->exprs, + relid, + NIL), + "stat", + namespaceId); + } + namestrcpy(&stxname, namestr); + + /* + * Check we have creation rights in target namespace. Skip check if + * caller doesn't want it. + */ + if (check_rights) + { + AclResult aclresult; + + aclresult = object_aclcheck(NamespaceRelationId, namespaceId, + GetUserId(), ACL_CREATE); + if (aclresult != ACLCHECK_OK) + aclcheck_error(aclresult, OBJECT_SCHEMA, + get_namespace_name(namespaceId)); + } + + /* + * Deal with the possibility that the statistics object already exists. + */ + if (SearchSysCacheExists2(STATEXTNAMENSP, + CStringGetDatum(namestr), + ObjectIdGetDatum(namespaceId))) + { + if (stmt->if_not_exists) + { + /* + * Since stats objects aren't members of extensions (see comments + * below), no need for checkMembershipInCurrentExtension here. + */ + ereport(NOTICE, + (errcode(ERRCODE_DUPLICATE_OBJECT), + errmsg("statistics object \"%s\" already exists, skipping", + namestr))); + /* Close relations */ + relation_close(rel, NoLock); + foreach(lc, other_rels) + relation_close((Relation) lfirst(lc), NoLock); + return InvalidObjectAddress; + } + + ereport(ERROR, + (errcode(ERRCODE_DUPLICATE_OBJECT), + errmsg("statistics object \"%s\" already exists", namestr))); + } + /* * Check that at least two columns were specified in the statement, or * that we're building statistics on a single expression (or virtual - * generated column). + * generated column). Join statistics are exempt since single-column MCV + * over a join captures cross-relation frequency data. */ - if (numcols == 1) + if (numcols == 1 && !isjoin) { Node *single = (Node *) linitial(stxexprs); @@ -431,9 +513,12 @@ CreateStatistics(List *relids, CreateStatsStmt *stmt, bool check_rights) /* * If no statistic type was specified, build them all (but only when the - * statistics is defined on more than one column/expression). + * statistics is defined on more than one column/expression). Join + * statistics support only MCV lists, on any number of columns. */ - if ((!requested_type) && (numcols >= 2)) + if (!requested_type && isjoin) + build_mcv = true; + else if (!requested_type && numcols >= 2) { build_ndistinct = true; build_dependencies = true; @@ -447,18 +532,22 @@ CreateStatistics(List *relids, CreateStatsStmt *stmt, bool check_rights) * like expressions they get no pg_statistic row, so they need * per-expression stats). Plain columns are skipped, since they already * have univariate statistics in pg_statistic. These give good estimates - * where per-clause estimates matter (e.g. functional dependencies). + * where per-clause estimates matter (e.g. functional dependencies). Join + * statistics cover plain columns only, so they build none. */ build_expressions = false; - foreach(cell, stxexprs) + if (!isjoin) { - Node *expr = (Node *) lfirst(cell); - - if (!IsA(expr, Var) || - get_attgenerated(relid, ((Var *) expr)->varattno) == ATTRIBUTE_GENERATED_VIRTUAL) + foreach(cell, stxexprs) { - build_expressions = true; - break; + Node *expr = (Node *) lfirst(cell); + + if (!IsA(expr, Var) || + get_attgenerated(relid, ((Var *) expr)->varattno) == ATTRIBUTE_GENERATED_VIRTUAL) + { + build_expressions = true; + break; + } } } @@ -503,6 +592,15 @@ CreateStatistics(List *relids, CreateStatsStmt *stmt, bool check_rights) errmsg("duplicate expression in statistics definition"))); } + /* + * For join statistics, only MCV is currently supported. + */ + if (isjoin && (build_ndistinct || build_dependencies || build_expressions)) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("only MCV statistics are supported for join statistics"), + errhint("ndistinct, dependencies, and expression statistics require a single table."))); + /* construct the char array of enabled statistic types */ ntypes = 0; if (build_ndistinct) @@ -540,6 +638,35 @@ CreateStatistics(List *relids, CreateStatsStmt *stmt, bool check_rights) values[Anum_pg_statistic_ext_stxkind - 1] = PointerGetDatum(stxkind); values[Anum_pg_statistic_ext_stxexprs - 1] = exprsDatum; + /* + * For join statistics, populate stxjoinrels and stxjoinconds. For + * single-table statistics, these fields are NULL. + */ + if (isjoin) + { + int njoinrels = list_length(stmt->stxjoinrels); + Oid *joinrel_arr; + int idx; + + /* stxjoinrels: anchor OID followed by the other relation OIDs */ + joinrel_arr = palloc(njoinrels * sizeof(Oid)); + idx = 0; + foreach(lc, stmt->stxjoinrels) + joinrel_arr[idx++] = lfirst_oid(lc); + values[Anum_pg_statistic_ext_stxjoinrels - 1] = + PointerGetDatum(buildoidvector(joinrel_arr, njoinrels)); + + /* stxjoinconds: serialize the join conditions */ + values[Anum_pg_statistic_ext_stxjoinconds - 1] = + CStringGetTextDatum(nodeToString(stmt->stxjoinconds)); + } + else + { + /* Join fields are NULL for single-table statistics */ + nulls[Anum_pg_statistic_ext_stxjoinrels - 1] = true; + nulls[Anum_pg_statistic_ext_stxjoinconds - 1] = true; + } + /* insert it into pg_statistic_ext */ htup = heap_form_tuple(statrel->rd_att, values, nulls); CatalogTupleInsert(statrel, htup); @@ -556,11 +683,16 @@ CreateStatistics(List *relids, CreateStatsStmt *stmt, bool check_rights) InvokeObjectPostCreateHook(StatisticExtRelationId, statoid, 0); /* - * Invalidate relcache so that others see the new statistics object. + * Invalidate relcache so that others see the new statistics object. Only + * the anchor relation needs invalidation, as planning finds the object + * through its statlist. */ CacheInvalidateRelcache(rel); + /* Close relations */ relation_close(rel, NoLock); + foreach(lc, other_rels) + relation_close((Relation) lfirst(lc), NoLock); /* * Add an AUTO dependency on each column used in the stats, so that the @@ -568,6 +700,10 @@ CreateStatistics(List *relids, CreateStatsStmt *stmt, bool check_rights) */ ObjectAddressSet(myself, StatisticExtRelationId, statoid); + /* For join statistics, record dependencies on the columns of every table */ + if (isjoin) + record_join_stats_dependencies(&myself, stmt, stxexprs); + /* * If there are no dependencies on a column, give the statistics object an * auto dependency on the whole table. In most cases, this will be @@ -587,16 +723,22 @@ CreateStatistics(List *relids, CreateStatsStmt *stmt, bool check_rights) /* * Store dependencies on anything mentioned in statistics expressions, - * just like we do for index expressions. + * just like we do for index expressions. For join stats, the explicit + * dependency recording above handles columns across multiple relations; + * recordDependencyOnSingleRelExpr only works for a single relation's + * Vars. */ - if (check_rights) - CheckUsageOnTypesInSingleRelExpr((Node *) stxexprs, relid, GetUserId()); - - recordDependencyOnSingleRelExpr(&myself, - (Node *) stxexprs, - relid, - DEPENDENCY_NORMAL, - DEPENDENCY_AUTO, false); + if (!isjoin) + { + if (check_rights) + CheckUsageOnTypesInSingleRelExpr((Node *) stxexprs, relid, GetUserId()); + + recordDependencyOnSingleRelExpr(&myself, + (Node *) stxexprs, + relid, + DEPENDENCY_NORMAL, + DEPENDENCY_AUTO, false); + } /* * Also add dependencies on namespace and owner. These are required @@ -820,6 +962,230 @@ RemoveStatisticsById(Oid statsOid) table_close(relation, RowExclusiveLock); } +/* + * check_join_stats_rel - + * Check that a relation can take part in join statistics. + */ +static void +check_join_stats_rel(Relation rel) +{ + /* + * XXX Partitioned tables are not supported yet: ANALYZE cannot probe one + * through its index, and the planner does not choose between join + * statistics with and without the partitions. + */ + if (rel->rd_rel->relkind != RELKIND_RELATION && + rel->rd_rel->relkind != RELKIND_MATVIEW && + rel->rd_rel->relkind != RELKIND_FOREIGN_TABLE) + ereport(ERROR, + (errcode(ERRCODE_WRONG_OBJECT_TYPE), + errmsg("cannot define statistics for relation \"%s\"", + RelationGetRelationName(rel)), + errdetail_relkind_not_supported(rel->rd_rel->relkind))); + + /* + * XXX Nor are tables with inheritance children, which the sampling would + * not cover. find_inheritance_children() rather than has_subclass(), + * which stays true after the last child is dropped. + */ + if (find_inheritance_children(RelationGetRelid(rel), NoLock) != NIL) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("join statistics do not support tables with inheritance children"), + errdetail("Table \"%s\" has inheritance children.", + RelationGetRelationName(rel)))); + + if (IsConflictLogTableClass(rel->rd_rel)) + ereport(ERROR, + (errcode(ERRCODE_WRONG_OBJECT_TYPE), + errmsg("cannot create statistics on conflict log table \"%s\"", + RelationGetRelationName(rel)), + errdetail("Conflict log tables are system-managed tables for logical replication conflicts."))); + + if (!allowSystemTableMods && IsSystemRelation(rel)) + ereport(ERROR, + (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE), + errmsg("permission denied: \"%s\" is a system catalog", + RelationGetRelationName(rel)))); +} + +/* + * open_join_stats_rels - + * Open the relations of a join statistics object and check that each + * can take part in one. + * + * Returns the tables after the anchor in stxjoinrels order, the anchor in + * *anchor_rel. The owner needs to own the anchor and have SELECT on the + * other tables, except when check_rights is false: ALTER TABLE re-creating + * the object for its existing owner. NoLock, as parse analysis of the + * statement locked the tables. + */ +static List * +open_join_stats_rels(CreateStatsStmt *stmt, Oid stxowner, bool check_rights, + Relation *anchor_rel) +{ + Relation rel; + List *other_rels = NIL; + ListCell *lc; + + rel = relation_open(stmt->stxrelid, NoLock); + check_join_stats_rel(rel); + if (check_rights && + !object_ownercheck(RelationRelationId, RelationGetRelid(rel), stxowner)) + aclcheck_error(ACLCHECK_NOT_OWNER, + get_relkind_objtype(rel->rd_rel->relkind), + RelationGetRelationName(rel)); + *anchor_rel = rel; + + foreach(lc, stmt->stxjoinrels) + { + Relation jrel; + AclResult aclresult; + + /* stxjoinrels[0] is the anchor, opened above */ + if (foreach_current_index(lc) == 0) + continue; + + jrel = relation_open(lfirst_oid(lc), NoLock); + check_join_stats_rel(jrel); + + /* Joined relations need only SELECT */ + if (check_rights) + { + aclresult = pg_class_aclcheck(RelationGetRelid(jrel), stxowner, + ACL_SELECT); + if (aclresult != ACLCHECK_OK) + aclcheck_error(aclresult, + get_relkind_objtype(jrel->rd_rel->relkind), + RelationGetRelationName(jrel)); + } + + other_rels = lappend(other_rels, jrel); + } + + return other_rels; +} + +/* + * check_join_stats_indexes - + * Warn about a join condition whose probed table has no index to + * sample with. + * + * Sampling probes the later table of each condition from the earlier one, + * so that table needs an index on its join column. The object is created + * anyway: ANALYZE skips it while there is no index and uses any suitable + * one it finds, so no dependency on an index is recorded. + */ +static void +check_join_stats_indexes(CreateStatsStmt *stmt, List *other_rels) +{ + ListCell *lc; + + foreach(lc, stmt->stxjoinconds) + { + OpExpr *opexpr = lfirst_node(OpExpr, lc); + Var *sampled_var; + Var *indexed_var; + bool commuted; + Relation indexed_rel; + + join_stats_cond_sides(opexpr, &sampled_var, &indexed_var, &commuted); + + /* other_rels starts at varno 2, as the anchor is never probed */ + Assert(indexed_var->varno > 1); + indexed_rel = (Relation) list_nth(other_rels, indexed_var->varno - 2); + + if (!OidIsValid(find_index_for_operator(indexed_rel, + indexed_var->varattno, + opexpr->opno))) + ereport(WARNING, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("no suitable index on \"%s\" column \"%s\" for join statistics", + RelationGetRelationName(indexed_rel), + get_attname(RelationGetRelid(indexed_rel), + indexed_var->varattno, false)), + errdetail("ANALYZE skips the statistics object while the table has no suitable index."), + errhint("Create an index on the join column to enable index-based join sampling."))); + } +} + +/* + * check_join_stats_columns - + * Refuse the columns join statistics cannot be built on. + */ +static void +check_join_stats_columns(CreateStatsStmt *stmt, List *stxexprs) +{ + ListCell *lc; + + foreach(lc, stxexprs) + { + Var *var = lfirst_node(Var, lc); + Oid key_relid; + + key_relid = list_nth_oid(stmt->stxjoinrels, var->varno - 1); + + if (get_attgenerated(key_relid, var->varattno) == ATTRIBUTE_GENERATED_VIRTUAL) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("statistics creation on virtual generated columns is not supported"))); + } +} + +/* + * record_join_stats_dependencies - + * Record AUTO dependencies on the columns a join statistics object + * uses, in its join conditions and its column list. + * + * recordDependencyOnSingleRelExpr() cannot be used, as the Vars belong to + * several relations. Join statistics have no expressions, so there are no + * other objects to depend on. + */ +static void +record_join_stats_dependencies(const ObjectAddress *myself, + CreateStatsStmt *stmt, List *stxexprs) +{ + ObjectAddresses *addrs; + ObjectAddress referenced; + ListCell *lc; + + addrs = new_object_addresses(); + + foreach(lc, stmt->stxjoinconds) + { + Var *vars[2]; + + if (!join_stats_cond_vars(lfirst(lc), &vars[0], &vars[1])) + elog(ERROR, "unexpected join statistics condition"); + for (int j = 0; j < 2; j++) + { + /* varno N is stxjoinrels[N-1] (varno 1 is the anchor) */ + Assert(vars[j]->varno >= 1 && + vars[j]->varno <= list_length(stmt->stxjoinrels)); + ObjectAddressSubSet(referenced, RelationRelationId, + list_nth_oid(stmt->stxjoinrels, + vars[j]->varno - 1), + vars[j]->varattno); + add_exact_object_address(&referenced, addrs); + } + } + + foreach(lc, stxexprs) + { + Var *var = lfirst_node(Var, lc); + + Assert(var->varno >= 1 && + var->varno <= list_length(stmt->stxjoinrels)); + ObjectAddressSubSet(referenced, RelationRelationId, + list_nth_oid(stmt->stxjoinrels, var->varno - 1), + var->varattno); + add_exact_object_address(&referenced, addrs); + } + + record_object_address_dependencies(myself, addrs, DEPENDENCY_AUTO); + free_object_addresses(addrs); +} + /* * Select a nonconflicting name for a new statistics object. * @@ -872,6 +1238,12 @@ ChooseExtendedStatisticName(const char *name1, const char *name2, * names for it. This will be passed to ChooseExtendedStatisticName along * with the parent table name and a suitable label. * + * For a join statistics object (joinrels != NIL) the columns are qualified + * with their relation name and the result is used as "name1" instead, since a + * bare column name would not identify which relation it came from. The + * relation name is written only when it differs from the previous column's, + * so (t1.a, t1.b, t2.c, t2.d) gives t1_a_b_t2_c_d. + * * We know that less than NAMEDATALEN characters will actually be used, * so we can truncate the result once we've generated that many. * @@ -879,16 +1251,18 @@ ChooseExtendedStatisticName(const char *name1, const char *name2, * ChooseIndexNameAddition. */ static char * -ChooseExtendedStatisticNameAddition(List *exprs, Oid relid) +ChooseExtendedStatisticNameAddition(List *exprs, Oid relid, List *joinrels) { char buf[NAMEDATALEN * 2]; int buflen = 0; + Oid prevrelid = InvalidOid; ListCell *lc; buf[0] = '\0'; foreach(lc, exprs) { StatsElem *selem = (StatsElem *) lfirst(lc); + char qualbuf[NAMEDATALEN * 2]; const char *name; /* It should be one of these, but just skip if it happens not to be */ @@ -899,10 +1273,33 @@ ChooseExtendedStatisticNameAddition(List *exprs, Oid relid) /* * A simple column reference resolves to a bare Var; use the column - * name. + * name. For a join object, qualify it with the relation name when + * the relation differs from the previous column's. */ if (!name && selem->expr && IsA(selem->expr, Var)) - name = get_attname(relid, ((Var *) selem->expr)->varattno, true); + { + Var *var = (Var *) selem->expr; + Oid colrelid = joinrels ? list_nth_oid(joinrels, var->varno - 1) : relid; + + name = get_attname(colrelid, var->varattno, true); + + if (name && joinrels && colrelid != prevrelid) + { + char *relname = get_rel_name(colrelid); + + if (relname) + { + snprintf(qualbuf, sizeof(qualbuf), "%s_%s", relname, name); + name = qualbuf; + } + else + name = NULL; /* defensive: fall back to "expr" */ + } + + prevrelid = name ? colrelid : InvalidOid; + } + else + prevrelid = InvalidOid; /* * We use fixed 'expr' for expressions, which have empty column names. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index b6700929459..9c0adfc4671 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -484,6 +484,7 @@ static void ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd, AlterTableUtilityContext *context); static void ATRewriteCatalogs(List **wqueue, LOCKMODE lockmode, AlterTableUtilityContext *context); +static bool ATOnlyRebuildsStatistics(AlteredTableInfo *tab); static void ATExecCmd(List **wqueue, AlteredTableInfo *tab, AlterTableCmd *cmd, LOCKMODE lockmode, AlterTablePass cur_pass, AlterTableUtilityContext *context); @@ -5440,6 +5441,10 @@ ATRewriteCatalogs(List **wqueue, LOCKMODE lockmode, { AlteredTableInfo *tab = (AlteredTableInfo *) lfirst(ltab); + /* a table whose join statistics alone were re-created was not altered */ + if (ATOnlyRebuildsStatistics(tab)) + continue; + /* * If the table is source table of ATTACH PARTITION command, we did * not modify anything about it that will change its toasting @@ -5453,6 +5458,37 @@ ATRewriteCatalogs(List **wqueue, LOCKMODE lockmode, } } +/* + * ATOnlyRebuildsStatistics: is re-creating statistics objects the only work + * queued for this table? + * + * This is the case for the table join statistics belong to, when the type of + * a column of another of the joined tables is changed. The table itself was + * not altered, and is locked only with ShareUpdateExclusiveLock (see + * ATPostAlterTypeCleanup), so the toast table check must skip it rather + * than lock it more strongly. + */ +static bool +ATOnlyRebuildsStatistics(AlteredTableInfo *tab) +{ + bool found = false; + + if (tab->rewrite != 0) + return false; + + for (AlterTablePass pass = 0; pass < AT_NUM_PASSES; pass++) + { + foreach_node(AlterTableCmd, cmd, tab->subcmds[pass]) + { + if (cmd->subtype != AT_ReAddStatistics) + return false; + found = true; + } + } + + return found; +} + /* * ATExecCmd: dispatch a subcommand to appropriate execution routine */ diff --git a/src/backend/optimizer/path/clausesel.c b/src/backend/optimizer/path/clausesel.c index c22cbfc9ca0..0938158e741 100644 --- a/src/backend/optimizer/path/clausesel.c +++ b/src/backend/optimizer/path/clausesel.c @@ -966,6 +966,15 @@ clause_selectivity_ext(PlannerInfo *root, s1 = boolvarsel(root, clause, varRelid); } + /* + * A clause on varRelid that also mentions other relations (hence not + * cacheable) is a join clause applied by a parameterized scan of + * varRelid. Correct its estimate with join statistics, as the join step + * would (see join_mcv_restriction_selectivity()). + */ + if (use_extended_stats && rinfo != NULL && !cacheable && varRelid != 0) + s1 = join_mcv_restriction_selectivity(root, rinfo, varRelid, s1); + /* Cache the result if possible */ if (cacheable) { diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c index f760ef9aa3a..1fc720d9d5d 100644 --- a/src/backend/optimizer/path/costsize.c +++ b/src/backend/optimizer/path/costsize.c @@ -105,6 +105,7 @@ #include "optimizer/plancat.h" #include "optimizer/restrictinfo.h" #include "parser/parsetree.h" +#include "statistics/statistics.h" #include "utils/lsyscache.h" #include "utils/selfuncs.h" #include "utils/spccache.h" @@ -5767,9 +5768,11 @@ calc_joinrel_size_estimate(PlannerInfo *root, List *restrictlist) { JoinType jointype = sjinfo->jointype; + List *joinclauses = restrictlist; Selectivity fkselec; Selectivity jselec; Selectivity pselec; + Selectivity selec; /* of all the join clauses, corrected */ double nrows; /* @@ -5841,6 +5844,14 @@ calc_joinrel_size_estimate(PlannerInfo *root, pselec = 0.0; /* not used, keep compiler quiet */ } + selec = fkselec * jselec; + if (jointype != JOIN_SEMI && jointype != JOIN_ANTI) + selec = join_mcv_join_selectivity(root, + outer_rel->relids, + inner_rel->relids, + joinclauses, + selec); + /* * Basically, we multiply size of Cartesian product by selectivity. * @@ -5856,17 +5867,17 @@ calc_joinrel_size_estimate(PlannerInfo *root, switch (jointype) { case JOIN_INNER: - nrows = outer_rows * inner_rows * fkselec * jselec; + nrows = outer_rows * inner_rows * selec; /* pselec not used */ break; case JOIN_LEFT: - nrows = outer_rows * inner_rows * fkselec * jselec; + nrows = outer_rows * inner_rows * selec; if (nrows < outer_rows) nrows = outer_rows; nrows *= pselec; break; case JOIN_FULL: - nrows = outer_rows * inner_rows * fkselec * jselec; + nrows = outer_rows * inner_rows * selec; if (nrows < outer_rows) nrows = outer_rows; if (nrows < inner_rows) @@ -5874,11 +5885,11 @@ calc_joinrel_size_estimate(PlannerInfo *root, nrows *= pselec; break; case JOIN_SEMI: - nrows = outer_rows * fkselec * jselec; + nrows = outer_rows * selec; /* pselec not used */ break; case JOIN_ANTI: - nrows = outer_rows * (1.0 - fkselec * jselec); + nrows = outer_rows * (1.0 - selec); nrows *= pselec; break; default: diff --git a/src/backend/optimizer/path/equivclass.c b/src/backend/optimizer/path/equivclass.c index 439cb651c01..0aa8b5382c9 100644 --- a/src/backend/optimizer/path/equivclass.c +++ b/src/backend/optimizer/path/equivclass.c @@ -3530,6 +3530,85 @@ have_relevant_eclass_joinclause(PlannerInfo *root, } +/* + * count_relevant_eclass_joinclauses + * Count the joinclauses EquivalenceClasses could produce between the + * two given relations. + * + * Like have_relevant_eclass_joinclause(), but counts them. As there, we + * don't check that each could actually be generated, so the count may be + * too high. + */ +int +count_relevant_eclass_joinclauses(PlannerInfo *root, + RelOptInfo *rel1, RelOptInfo *rel2) +{ + Bitmapset *matching_ecs; + Relids join_relids; + Relids nominal_relids1; + Relids nominal_relids2; + int count = 0; + int i; + + /* Examine only eclasses mentioning both rel1 and rel2 */ + matching_ecs = get_common_eclass_indexes(root, rel1->relids, + rel2->relids); + + join_relids = bms_union(rel1->relids, rel2->relids); + + /* Source clauses of a broken EC refer to the top parents of child rels */ + nominal_relids1 = IS_OTHER_REL(rel1) ? rel1->top_parent_relids : rel1->relids; + nominal_relids2 = IS_OTHER_REL(rel2) ? rel2->top_parent_relids : rel2->relids; + + i = -1; + while ((i = bms_next_member(matching_ecs, i)) >= 0) + { + EquivalenceClass *ec = (EquivalenceClass *) list_nth(root->eq_classes, + i); + EquivalenceMemberIterator it; + EquivalenceMember *em; + + /* A single-member EC generates no joinclauses */ + if (list_length(ec->ec_members) <= 1) + continue; + + /* + * An EC with a constant generates restriction clauses instead. A + * member using both relations does yield a joinclause then, but it is + * distributed to the joininfo lists and counted there. + */ + if (ec->ec_has_const) + continue; + + if (ec->ec_broken) + { + foreach_node(RestrictInfo, rinfo, ec->ec_sources) + { + if (bms_overlap(nominal_relids1, rinfo->clause_relids) && + bms_overlap(nominal_relids2, rinfo->clause_relids)) + count++; + } + continue; + } + + count++; + + /* Each member using both relations is equated by one more clause */ + setup_eclass_member_iterator(&it, ec, join_relids); + while ((em = eclass_member_iterator_next(&it)) != NULL) + { + if (bms_overlap(em->em_relids, rel1->relids) && + bms_overlap(em->em_relids, rel2->relids)) + count++; + } + } + + bms_free(join_relids); + + return count; +} + + /* * has_relevant_eclass_joinclause * Detect whether there is an EquivalenceClass that could produce diff --git a/src/backend/optimizer/util/joininfo.c b/src/backend/optimizer/util/joininfo.c index f75e35f22ae..ecd630bbf97 100644 --- a/src/backend/optimizer/util/joininfo.c +++ b/src/backend/optimizer/util/joininfo.c @@ -81,6 +81,56 @@ have_relevant_joinclause(PlannerInfo *root, } +/* + * count_relevant_joinclauses + * Count the joinclauses that reference both given relations. + * + * Like have_relevant_joinclause(), but counts them. A clause that also + * references other relations is counted too, so the count may be too high, + * but never too low: callers use it to rule out joins on more than one + * clause. + */ +int +count_relevant_joinclauses(PlannerInfo *root, + RelOptInfo *rel1, RelOptInfo *rel2) +{ + int count = 0; + ListCell *l; + + /* a clause involving both relations is in both joininfo lists */ + foreach(l, rel1->joininfo) + { + RestrictInfo *rinfo = (RestrictInfo *) lfirst(l); + + /* + * If the join commutes with some other outer join(s) via outer join + * identity 3, there will be multiple clones of its join clauses in + * the joininfo list. Count only the has_clone form of such clauses. + */ + if (rinfo->is_clone) + continue; + + /* + * A clause on one relation in an outer join's ON clause is in both + * joininfo lists, as it is evaluated at the join, but does not + * reference both. + */ + if (bms_overlap(rel1->relids, rinfo->clause_relids) && + bms_overlap(rel2->relids, rinfo->clause_relids)) + count++; + } + + /* + * We also need to count from the EquivalenceClass data structure, which + * might contain relationships not emitted into the joininfo lists. + */ + if (rel1->has_eclass_joins && rel2->has_eclass_joins) + count += count_relevant_eclass_joinclauses(root, rel1, rel2); + + return count; +} + + /* * add_join_clause_to_rels * Add 'restrictinfo' to the joininfo list of each relation it requires. diff --git a/src/backend/optimizer/util/plancat.c b/src/backend/optimizer/util/plancat.c index f7d4454f232..6b4ae365851 100644 --- a/src/backend/optimizer/util/plancat.c +++ b/src/backend/optimizer/util/plancat.c @@ -1655,7 +1655,8 @@ get_relation_constraints(PlannerInfo *root, static void get_relation_statistics_worker(List **stainfos, RelOptInfo *rel, Oid statOid, bool inh, - List *exprs, Bitmapset *columns) + List *exprs, Bitmapset *columns, + List *joinrels, List *joinconds) { Form_pg_statistic_ext_data dataForm; HeapTuple dtup; @@ -1707,6 +1708,14 @@ get_relation_statistics_worker(List **stainfos, RelOptInfo *rel, info->exprs = exprs; info->columns = columns; + /* + * Only MCV carries the join fields: CREATE STATISTICS allows no other + * kind yet for join stats, so the nodes above are never join stats. + * For single-table stats these locals are NIL. + */ + info->joinrels = joinrels; + info->joinconds = joinconds; + *stainfos = lappend(*stainfos, info); } @@ -1752,6 +1761,11 @@ get_relation_statistics(PlannerInfo *root, RelOptInfo *rel, HeapTuple htup; List *exprs = NIL; Bitmapset *columns = NULL; + List *joinrels = NIL; + List *joinconds = NIL; + bool isjoin; + bool isnull; + Datum datum; htup = SearchSysCache1(STATEXTOID, ObjectIdGetDatum(statOid)); if (!HeapTupleIsValid(htup)) @@ -1768,6 +1782,26 @@ get_relation_statistics(PlannerInfo *root, RelOptInfo *rel, */ exprs = statext_get_stxexprs(htup, relation, &columns); + datum = SysCacheGetAttr(STATEXTOID, htup, + Anum_pg_statistic_ext_stxjoinrels, &isnull); + isjoin = !isnull; + if (isjoin) + { + oidvector *jrels = (oidvector *) DatumGetPointer(datum); + char *condstr; + + for (int j = 0; j < jrels->dim1; j++) + joinrels = lappend_oid(joinrels, jrels->values[j]); + + datum = SysCacheGetAttrNotNull(STATEXTOID, htup, + Anum_pg_statistic_ext_stxjoinconds); + condstr = TextDatumGetCString(datum); + joinconds = (List *) stringToNode(condstr); + pfree(condstr); + + root->hasJoinStats = true; + } + /* * Modify the copies we obtain from the relcache to have the correct * varno for the parent relation, so that they match up correctly @@ -1775,8 +1809,10 @@ get_relation_statistics(PlannerInfo *root, RelOptInfo *rel, * * This must be done before const-simplification because * eval_const_expressions reduces NullTest for Vars based on varno. + * + * Join statistics keep their varnos, which index stxjoinrels. */ - if (varno != 1) + if (varno != 1 && !isjoin) ChangeVarNodes((Node *) exprs, 1, varno, 0); /* @@ -1797,10 +1833,10 @@ get_relation_statistics(PlannerInfo *root, RelOptInfo *rel, /* extract statistics for possible values of stxdinherit flag */ get_relation_statistics_worker(&stainfos, rel, statOid, true, - exprs, columns); + exprs, columns, joinrels, joinconds); get_relation_statistics_worker(&stainfos, rel, statOid, false, - exprs, columns); + exprs, columns, joinrels, joinconds); ReleaseSysCache(htup); } diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 0563453fe24..52e40c8e1d5 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -4841,6 +4841,13 @@ stats_param: ColId $$->name = $1; $$->expr = NULL; } + | ColId indirection + { + /* Table-qualified column reference (e.g., tbl.col) */ + $$ = makeNode(StatsElem); + $$->name = NULL; + $$->expr = (Node *) makeColumnRef($1, $2, @1, yyscanner); + } | func_expr_windowless { $$ = makeNode(StatsElem); diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 76e85b8898c..52438aab7f7 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -59,7 +59,9 @@ #include "parser/parse_type.h" #include "parser/parse_utilcmd.h" #include "parser/parser.h" +#include "parser/parsetree.h" #include "rewrite/rewriteManip.h" +#include "statistics/statistics.h" #include "utils/acl.h" #include "utils/builtins.h" #include "utils/lsyscache.h" @@ -1621,6 +1623,8 @@ expandTableLikeClause(RangeVar *heapRel, TableLikeClause *table_like_clause) RelationGetRelid(childrel), parent_stat_oid, attmap); + if (stats_stmt == NULL) + continue; /* Copy comment on statistics object, if requested */ if (table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) @@ -2066,6 +2070,10 @@ generateClonedIndexStmt(RangeVar *heapRel, Relation source_idx, * relation. Those attnums, along with the attribute numbers referenced by * Vars inside complex expressions, are remapped to the new relation's * numbering according to attmap. + * + * Returns NULL for a join statistics object: it describes a join of the + * parent relation with other relations, not the parent relation itself, so + * there is nothing to copy to the new relation. */ static CreateStatsStmt * generateClonedExtStatsStmt(RangeVar *heapRel, Oid heapRelid, @@ -2076,6 +2084,7 @@ generateClonedExtStatsStmt(RangeVar *heapRel, Oid heapRelid, List *stat_types = NIL; List *def_names = NIL; Datum datum; + bool isnull; ArrayType *arr; char *enabled; int i; @@ -2093,6 +2102,15 @@ generateClonedExtStatsStmt(RangeVar *heapRel, Oid heapRelid, if (!HeapTupleIsValid(ht_stats)) elog(ERROR, "cache lookup failed for statistics object %u", source_statsid); + /* Join statistics are not copied */ + (void) SysCacheGetAttr(STATEXTOID, ht_stats, + Anum_pg_statistic_ext_stxjoinrels, &isnull); + if (!isnull) + { + ReleaseSysCache(ht_stats); + return NULL; + } + /* Determine which statistics types exist */ datum = SysCacheGetAttrNotNull(STATEXTOID, ht_stats, Anum_pg_statistic_ext_stxkind); @@ -3169,12 +3187,183 @@ transformIndexStmt(Oid relid, IndexStmt *stmt, const char *queryString) return stmt; } +/* + * lock_join_stats_rels - lock the tables of a join statistics FROM clause + */ +static List * +lock_join_stats_rels(Node *node, List *relids) +{ + JoinExpr *j; + + if (IsA(node, RangeVar)) + return lappend_oid(relids, + RangeVarGetRelid((RangeVar *) node, + ShareUpdateExclusiveLock, false)); + if (!IsA(node, JoinExpr)) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("unsupported FROM clause element in join statistics"))); + + j = (JoinExpr *) node; + relids = lock_join_stats_rels(j->larg, relids); + return lock_join_stats_rels(j->rarg, relids); +} + +/* + * collect_join_stats_tree - collect the tables and conditions of a + * transformed join tree + */ +static void +collect_join_stats_tree(ParseState *pstate, Node *node, + List **rtindexes, List **conds) +{ + JoinExpr *j; + + if (IsA(node, RangeTblRef)) + { + int rtindex = ((RangeTblRef *) node)->rtindex; + + /* lock_join_stats_rels() let through only table names and joins */ + if (rt_fetch(rtindex, pstate->p_rtable)->rtekind != RTE_RELATION) + elog(ERROR, "unexpected rtekind: %d", + (int) rt_fetch(rtindex, pstate->p_rtable)->rtekind); + *rtindexes = lappend_int(*rtindexes, rtindex); + return; + } + if (!IsA(node, JoinExpr)) + elog(ERROR, "unrecognized node type: %d", (int) nodeTag(node)); + + j = (JoinExpr *) node; + if (j->jointype != JOIN_INNER) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("join statistics are only supported for inner joins"))); + collect_join_stats_tree(pstate, j->larg, rtindexes, conds); + collect_join_stats_tree(pstate, j->rarg, rtindexes, conds); + if (j->quals) + { + /* transformFromClause() leaves collation assignment to its caller */ + assign_expr_collations(pstate, j->quals); + *conds = list_concat(*conds, make_ands_implicit((Expr *) j->quals)); + } +} + +/* mutator for renumber_join_stats_vars: renumber each Var's varno */ +static Node * +renumber_join_stats_vars_mutator(Node *node, int *map) +{ + Var *var; + + if (node == NULL) + return NULL; + if (!IsA(node, Var) || ((Var *) node)->varlevelsup != 0) + return expression_tree_mutator(node, renumber_join_stats_vars_mutator, + map); + + var = (Var *) copyObject(node); + var->varno = map[var->varno]; + var->varnosyn = var->varno; + return (Node *) var; +} + +/* + * renumber_join_stats_vars - renumber the Vars of a join statistics expression + * + * Replaces Vars that reference join outputs (e.g. a USING column) with the + * underlying table columns, then renumbers each Var's varno through map, + * which maps rangetable indexes to table positions (1 = anchor). + */ +static Node * +renumber_join_stats_vars(Query *query, Node *node, int *map) +{ + node = flatten_join_alias_for_parser(query, node, 0); + return renumber_join_stats_vars_mutator(node, map); +} + +/* + * is_join_stats_edge - is cond an equality between columns of two tables? + */ +static bool +is_join_stats_edge(Node *cond) +{ + Var *lvar; + Var *rvar; + + if (!join_stats_cond_vars(cond, &lvar, &rvar)) + return false; + return lvar->varno != rvar->varno && + AttrNumberIsForUserDefinedAttr(lvar->varattno) && + AttrNumberIsForUserDefinedAttr(rvar->varattno) && + get_mergejoin_opfamilies(((OpExpr *) cond)->opno) != NIL; +} + +/* + * order_join_stats_tables - canonical order of the tables of a join statistics + * object + * + * Numbers the tables breadth-first from the anchor (position 1) over the + * edges, breaking ties by FROM order, so that every table after the anchor is + * joined to an earlier one. Sets newpos[t] to the new position of the table + * at FROM position t (1..ntables). Returns 0, or the FROM position of the + * first table that no edge connects to the anchor. + */ +static int +order_join_stats_tables(List *edges, int ntables, int *newpos) +{ + int *order = palloc_array(int, ntables + 1); + int norder = 1; + int k; + int t; + ListCell *lc; + + memset(newpos, 0, (ntables + 1) * sizeof(int)); + order[1] = 1; + newpos[1] = 1; + for (k = 1; k <= norder; k++) + { + for (t = 1; t <= ntables; t++) + { + if (newpos[t] != 0) + continue; + foreach(lc, edges) + { + Var *lvar; + Var *rvar; + int a; + int b; + + if (!join_stats_cond_vars(lfirst(lc), &lvar, &rvar)) + elog(ERROR, "unexpected join statistics condition"); + a = lvar->varno; + b = rvar->varno; + if ((a == order[k] && b == t) || (b == order[k] && a == t)) + { + order[++norder] = t; + newpos[t] = norder; + break; + } + } + } + } + pfree(order); + + for (t = 1; t <= ntables; t++) + { + if (newpos[t] == 0) + return t; + } + return 0; +} + /* * transformStatsStmt - parse analysis for CREATE STATISTICS * - * To avoid race conditions, it's important that this function relies only on - * the passed-in relid (and not on stmt->relation) to determine the target - * relation. + * For single-relation statistics, to avoid race conditions, it's important + * that this function relies only on the passed-in relid (and not on + * stmt->relation) to determine the target relation. + * + * For join statistics, relid is InvalidOid and tables are resolved directly + * from stmt->relations. */ CreateStatsStmt * transformStatsStmt(Oid relid, CreateStatsStmt *stmt, const char *queryString) @@ -3182,7 +3371,11 @@ transformStatsStmt(Oid relid, CreateStatsStmt *stmt, const char *queryString) ParseState *pstate; ParseNamespaceItem *nsitem; ListCell *l; - Relation rel; + Relation rel = NULL; + bool isjoin = (stmt->relations != NIL && + IsA(linitial(stmt->relations), JoinExpr)); + int *rtmap = NULL; + Query *joinquery = NULL; /* Nothing to do if statement already transformed. */ if (stmt->transformed) @@ -3192,18 +3385,128 @@ transformStatsStmt(Oid relid, CreateStatsStmt *stmt, const char *queryString) pstate = make_parsestate(NULL); pstate->p_sourcetext = queryString; - /* - * Put the parent table into the rtable so that the expressions can refer - * to its fields without qualification. Caller is responsible for locking - * relation, but we still need to open it. - */ - rel = relation_open(relid, NoLock); - nsitem = addRangeTableEntryForRelation(pstate, rel, - AccessShareLock, - NULL, false, true); + if (isjoin) + { + List *lockedrelids; + ListCell *lc2; + List *rtindexes = NIL; + List *conds = NIL; + List *edges = NIL; + int ntables; + int *newpos; + Oid *relids_by_pos; + int unconnected; + int k; - /* no to join list, yes to namespaces */ - addNSItemToQuery(pstate, nsitem, false, true, true); + /* + * Parse the FROM clause as for a SELECT, so that names are resolved, + * and USING, NATURAL and parenthesized joins handled, the same way. + * Lock the tables first, so that the weaker lock + * transformFromClause() takes is not upgraded later, and check + * afterwards that each name still refers to the table we locked. + */ + lockedrelids = lock_join_stats_rels((Node *) linitial(stmt->relations), + NIL); + transformFromClause(pstate, stmt->relations); + /* a single JoinExpr in FROM yields a single jointree item */ + Assert(list_length(pstate->p_joinlist) == 1); + collect_join_stats_tree(pstate, (Node *) linitial(pstate->p_joinlist), + &rtindexes, &conds); + ntables = list_length(rtindexes); + Assert(ntables >= 2); + if (list_length(lockedrelids) != ntables) + elog(ERROR, "unexpected number of tables in join statistics"); + forboth(l, rtindexes, lc2, lockedrelids) + { + if (rt_fetch(lfirst_int(l), pstate->p_rtable)->relid != lfirst_oid(lc2)) + ereport(ERROR, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("table name \"%s\" was concurrently changed to refer to a different table", + get_rel_name(lfirst_oid(lc2))))); + } + + /* number the tables 1..ntables in FROM order */ + rtmap = palloc0_array(int, list_length(pstate->p_rtable) + 1); + foreach(l, rtindexes) + rtmap[lfirst_int(l)] = foreach_current_index(l) + 1; + joinquery = makeNode(Query); + joinquery->rtable = pstate->p_rtable; + + /* + * Every condition must be an equality between columns of two tables. + * Anything else, such as a condition on a single table, would make + * the statistics describe only the rows satisfying it. + */ + foreach(l, conds) + { + Node *cond = (Node *) lfirst(l); + Node *renumbered = renumber_join_stats_vars(joinquery, cond, + rtmap); + int ncondtables = bms_num_members(pull_varnos(NULL, renumbered)); + + if (ncondtables < 2 || !is_join_stats_edge(renumbered)) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("join statistics support only equality conditions between columns of two tables"), + ncondtables < 2 ? + errdetail("This condition does not refer to two tables.") : + errdetail("This condition is not an equality between two columns."), + parser_errposition(pstate, exprLocation(cond)))); + edges = lappend(edges, renumbered); + } + + /* + * The conditions must connect all the tables; a cross product cannot + * be sampled. Renumber the tables in canonical order, so that every + * table after the anchor is joined to an earlier one. + */ + newpos = palloc_array(int, ntables + 1); + unconnected = order_join_stats_tables(edges, ntables, newpos); + if (unconnected != 0) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("join conditions of join statistics must connect all the tables"), + errdetail("No join condition connects \"%s\" to \"%s\".", + rt_fetch(list_nth_int(rtindexes, unconnected - 1), + pstate->p_rtable)->eref->aliasname, + rt_fetch(linitial_int(rtindexes), + pstate->p_rtable)->eref->aliasname))); + /* From here on, rtmap gives canonical positions */ + foreach(l, rtindexes) + rtmap[lfirst_int(l)] = newpos[foreach_current_index(l) + 1]; + + /* + * stxjoinrels lists the tables in canonical order (stxjoinrels[0] == + * stxrelid, the anchor), so Var.varno N references stxjoinrels[N-1]. + */ + relids_by_pos = palloc_array(Oid, ntables + 1); + foreach(l, lockedrelids) + relids_by_pos[newpos[foreach_current_index(l) + 1]] = lfirst_oid(l); + for (k = 1; k <= ntables; k++) + stmt->stxjoinrels = lappend_oid(stmt->stxjoinrels, relids_by_pos[k]); + stmt->stxrelid = linitial_oid(stmt->stxjoinrels); + stmt->stxjoinconds = NIL; + foreach(l, conds) + stmt->stxjoinconds = lappend(stmt->stxjoinconds, + renumber_join_stats_vars(joinquery, + (Node *) lfirst(l), + rtmap)); + } + else + { + /* + * Put the parent table into the rtable so that the expressions can + * refer to its fields without qualification. Caller is responsible + * for locking relation, but we still need to open it. + */ + rel = relation_open(relid, NoLock); + nsitem = addRangeTableEntryForRelation(pstate, rel, + AccessShareLock, + NULL, false, true); + + /* no to join list, yes to namespaces */ + addNSItemToQuery(pstate, nsitem, false, true, true); + } /* take care of any expressions */ foreach(l, stmt->exprs) @@ -3218,22 +3521,30 @@ transformStatsStmt(Oid relid, CreateStatsStmt *stmt, const char *queryString) /* We have to fix its collations too */ assign_expr_collations(pstate, selem->expr); + + /* Join statistics columns use the canonical table positions */ + if (isjoin) + selem->expr = renumber_join_stats_vars(joinquery, selem->expr, + rtmap); } } - /* - * Check that only the base rel is mentioned. (This should be dead code - * now that add_missing_from is history.) - */ - if (list_length(pstate->p_rtable) != 1) - ereport(ERROR, - (errcode(ERRCODE_INVALID_COLUMN_REFERENCE), - errmsg("statistics expressions can refer only to the table being referenced"))); + if (!isjoin) + { + /* + * Check that only the base rel is mentioned. (This should be dead + * code now that add_missing_from is history.) + */ + if (list_length(pstate->p_rtable) != 1) + ereport(ERROR, + (errcode(ERRCODE_INVALID_COLUMN_REFERENCE), + errmsg("statistics expressions can refer only to the table being referenced"))); - free_parsestate(pstate); + /* Close relation */ + table_close(rel, NoLock); + } - /* Close relation */ - table_close(rel, NoLock); + free_parsestate(pstate); /* Mark statement as successfully transformed */ stmt->transformed = true; diff --git a/src/backend/statistics/Makefile b/src/backend/statistics/Makefile index 7ff5938b027..9639e0b95f5 100644 --- a/src/backend/statistics/Makefile +++ b/src/backend/statistics/Makefile @@ -17,6 +17,7 @@ OBJS = \ dependencies.o \ extended_stats.o \ extended_stats_funcs.o \ + join_mcv.o \ mcv.o \ mvdistinct.o \ relation_stats.o \ diff --git a/src/backend/statistics/extended_stats.c b/src/backend/statistics/extended_stats.c index 3080af0916a..6f26f68dfd5 100644 --- a/src/backend/statistics/extended_stats.c +++ b/src/backend/statistics/extended_stats.c @@ -21,6 +21,7 @@ #include "access/htup_details.h" #include "access/table.h" #include "catalog/indexing.h" +#include "catalog/pg_inherits.h" #include "catalog/pg_statistic_ext.h" #include "catalog/pg_statistic_ext_data.h" #include "commands/defrem.h" @@ -35,6 +36,7 @@ #include "rewrite/rewriteHandler.h" #include "statistics/extended_stats_internal.h" #include "statistics/statistics.h" +#include "storage/lmgr.h" #include "utils/acl.h" #include "utils/array.h" #include "utils/attoptcache.h" @@ -65,6 +67,7 @@ typedef struct StatExtEntry { Oid statOid; /* OID of pg_statistic_ext entry */ + Oid stxowner; /* statistics object's owner */ char *schema; /* statistics object's schema */ char *name; /* statistics object's name */ List *types; /* 'char' list of enabled statistics kinds */ @@ -72,6 +75,20 @@ typedef struct StatExtEntry List *exprs; /* all columns and expressions in * user-declared order, mirroring stxexprs */ Bitmapset *columns; /* positions in exprs of the plain columns */ + + /* + * Join statistics fields (NULL/invalid for single-table stats). + * + * For joins, the same attnum can appear from different relations, so we + * keep a per-column mapping of each covered column's attnum to its source + * relation varno. + */ + int16 *attnums; /* attribute numbers, one per stats column */ + int nattnums; /* length of attnums[] and attref_varnos[] */ + int16 *attref_varnos; /* source relation varno for each attnum */ + Oid *joinrels; /* participating relation OIDs, anchor first */ + int njoinrels; /* number of participating relations */ + List *joinconds; /* List of OpExpr: join conditions */ } StatExtEntry; @@ -171,40 +188,207 @@ BuildRelationExtStatistics(Relation onerel, bool inh, double totalrows, MCVList *mcv = NULL; Datum exprstats = (Datum) 0; VacAttrStats **stats; + VacAttrStats **mcv_stats = NULL; ListCell *lc2; int stattarget; StatsBuildData *data; + bool not_computed = false; + Oid join_index = InvalidOid; /* * Check if we can build these stats based on the column analyzed. If * not, report this fact (except in autovacuum) and move on. + * + * Join statistics are not checked here, as their columns can belong + * to other tables. */ stats = lookup_var_attr_stats(stat, natts, vacattrstats); - if (!stats) + if (!stats && stat->njoinrels == 0) { report_ext_stats_not_computed(onerel, stat); continue; } - /* compute statistics target for this statistics object */ - stattarget = statext_compute_stattarget(stat->stattarget, - list_length(stat->exprs), - stats); + /* check whether the join statistics can be built this time */ + if (stat->njoinrels > 0) + { + bool skip = false; + Var *sampled_var; + Var *probed_var; + bool commuted; + Relation probed_rel; + + /* tables with inheritance children are not supported yet */ + if (inh) + continue; - /* - * Don't rebuild statistics objects with statistics target set to 0 - * (we just leave the existing values around, just like we do for - * regular per-column statistics). - */ - if (stattarget == 0) - continue; + stattarget = stat->stattarget >= 0 ? stat->stattarget + : default_statistics_target; + + /* + * Don't rebuild statistics objects with statistics target set to + * 0 (we just leave the existing values around, just like we do + * for regular per-column statistics). + */ + if (stattarget == 0) + continue; + + /* + * Only a join of two tables on a single condition can be built so + * far, although CREATE STATISTICS accepts other shapes. Skip the + * others and keep any existing data. njoinrels counts the + * anchor. + */ + if (stat->njoinrels != 2 || list_length(stat->joinconds) != 1) + { + if (!AmAutoVacuumWorkerProcess()) + ereport(WARNING, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("skipping join statistics \"%s.%s\"", + stat->schema, stat->name), + errdetail("Statistics can currently be built only for joins of two tables on a single equality condition."), + errtable(onerel))); + continue; + } + + /* + * Refresh the statistics only if their owner, not the user + * running ANALYZE, still has SELECT on the other tables, and + * otherwise keep any existing data. The anchor is not checked, + * as for single-table statistics. + */ + for (int i = 1; i < stat->njoinrels; i++) + { + if (pg_class_aclcheck(stat->joinrels[i], stat->stxowner, + ACL_SELECT) != ACLCHECK_OK) + { + if (!AmAutoVacuumWorkerProcess()) + ereport(WARNING, + (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE), + errmsg("skipping join statistics \"%s.%s\"", + stat->schema, stat->name), + errdetail("Its owner lacks SELECT privilege on table \"%s.%s\".", + get_namespace_name(get_rel_namespace(stat->joinrels[i])), + get_rel_name(stat->joinrels[i])), + errtable(onerel))); + skip = true; + break; + } + } + if (skip) + continue; + + /* + * Don't wait for a lock on the other tables. ALTER TABLE on one + * of them holds AccessExclusiveLock while it waits for our lock + * on onerel to re-create these statistics, so waiting would + * deadlock. Skip the statistics instead and keep any existing + * data. + */ + for (int i = 1; i < stat->njoinrels; i++) + { + if (!ConditionalLockRelationOid(stat->joinrels[i], + AccessShareLock)) + { + if (!AmAutoVacuumWorkerProcess()) + ereport(WARNING, + (errcode(ERRCODE_LOCK_NOT_AVAILABLE), + errmsg("skipping join statistics \"%s.%s\"", + stat->schema, stat->name), + errdetail("Could not obtain lock on table \"%s.%s\".", + get_namespace_name(get_rel_namespace(stat->joinrels[i])), + get_rel_name(stat->joinrels[i])), + errtable(onerel))); + skip = true; + break; + } + } + if (skip) + continue; + + /* + * CREATE STATISTICS rejects tables with inheritance children, as + * the sampling does not cover them, but a table may have gained + * some since. Skip the statistics then and keep any existing + * data. + */ + for (int i = 0; i < stat->njoinrels; i++) + { + if (find_inheritance_children(stat->joinrels[i], NoLock) != NIL) + { + if (!AmAutoVacuumWorkerProcess()) + ereport(WARNING, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("skipping join statistics \"%s.%s\"", + stat->schema, stat->name), + errdetail("Table \"%s.%s\" has inheritance children.", + get_namespace_name(get_rel_namespace(stat->joinrels[i])), + get_rel_name(stat->joinrels[i])), + errtable(onerel))); + skip = true; + break; + } + } + if (skip) + continue; + + /* + * The join values are looked up in an index on the probed table's + * join column, so skip the statistics without one and keep any + * existing data. The lock taken above keeps the index from being + * dropped before the build. + * + * XXX Sampling by a sequential scan of the probed table, instead + * of index lookups, would remove this skip. + */ + join_stats_cond_sides(linitial_node(OpExpr, stat->joinconds), + &sampled_var, &probed_var, &commuted); + probed_rel = table_open(stat->joinrels[1], NoLock); + join_index = find_index_for_operator(probed_rel, + probed_var->varattno, + linitial_node(OpExpr, stat->joinconds)->opno); + if (!OidIsValid(join_index)) + { + if (!AmAutoVacuumWorkerProcess()) + ereport(WARNING, + (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE), + errmsg("skipping join statistics \"%s.%s\"", + stat->schema, stat->name), + errdetail("Table \"%s.%s\" has no suitable index on column \"%s\".", + get_namespace_name(RelationGetNamespace(probed_rel)), + RelationGetRelationName(probed_rel), + get_attname(RelationGetRelid(probed_rel), + probed_var->varattno, false)), + errtable(onerel))); + table_close(probed_rel, NoLock); + continue; + } + table_close(probed_rel, NoLock); - /* evaluate expressions (if the statistics object has any) */ - data = make_build_data(onerel, stat, numrows, rows, stats, stattarget); - if (!data) + data = NULL; + } + else { - report_ext_stats_not_computed(onerel, stat); - continue; + /* compute statistics target for this statistics object */ + stattarget = statext_compute_stattarget(stat->stattarget, + list_length(stat->exprs), + stats); + + /* + * Don't rebuild statistics objects with statistics target set to + * 0 (we just leave the existing values around, just like we do + * for regular per-column statistics). + */ + if (stattarget == 0) + continue; + + /* evaluate expressions (if the statistics object has any) */ + data = make_build_data(onerel, stat, numrows, rows, stats, stattarget); + if (!data) + { + report_ext_stats_not_computed(onerel, stat); + continue; + } } /* compute statistic of each requested type */ @@ -217,7 +401,25 @@ BuildRelationExtStatistics(Relation onerel, bool inh, double totalrows, else if (t == STATS_EXT_DEPENDENCIES) dependencies = statext_dependencies_build(data); else if (t == STATS_EXT_MCV) - mcv = statext_mcv_build(data, totalrows, stattarget); + { + if (stat->njoinrels > 0) + mcv = statext_join_mcv_build(onerel, + stat->joinrels, + stat->njoinrels, + stat->joinconds, + join_index, + stat->attnums, + stat->attref_varnos, + stat->nattnums, + stattarget, + numrows, rows, totalrows, + &mcv_stats); + else + { + mcv = statext_mcv_build(data, totalrows, stattarget); + mcv_stats = stats; + } + } else if (t == STATS_EXT_EXPRESSIONS) { AnlExprData *exprdata; @@ -252,7 +454,7 @@ BuildRelationExtStatistics(Relation onerel, bool inh, double totalrows, if (!exprdata) { report_ext_stats_not_computed(onerel, stat); - data = NULL; + not_computed = true; break; } nexprs = list_length(cmplxexprs); @@ -264,12 +466,13 @@ BuildRelationExtStatistics(Relation onerel, bool inh, double totalrows, } } - if (!data) + /* skip storing if an expression could not be analyzed */ + if (not_computed) continue; /* store the statistics in the catalog */ statext_store(stat->statOid, inh, - ndistinct, dependencies, mcv, exprstats, stats); + ndistinct, dependencies, mcv, exprstats, mcv_stats); /* for reporting progress */ pgstat_progress_update_param(PROGRESS_ANALYZE_EXT_STATS_COMPUTED, @@ -689,6 +892,7 @@ fetch_statentries_for_relation(Relation pg_statext, Relation rel) entry = palloc0_object(StatExtEntry); staForm = (Form_pg_statistic_ext) GETSTRUCT(htup); entry->statOid = staForm->oid; + entry->stxowner = staForm->stxowner; entry->schema = get_namespace_name(staForm->stxnamespace); entry->name = pstrdup(NameStr(staForm->stxname)); @@ -719,6 +923,71 @@ fetch_statentries_for_relation(Relation pg_statext, Relation rel) */ entry->exprs = statext_get_stxexprs(htup, rel, &entry->columns); + /* + * Fetch join statistics fields. These are all NULL for single-table + * statistics. + */ + datum = SysCacheGetAttr(STATEXTOID, htup, + Anum_pg_statistic_ext_stxjoinrels, &isnull); + if (!isnull) + { + oidvector *ov = (oidvector *) DatumGetPointer(datum); + + entry->njoinrels = ov->dim1; + entry->joinrels = (Oid *) palloc(ov->dim1 * sizeof(Oid)); + memcpy(entry->joinrels, ov->values, ov->dim1 * sizeof(Oid)); + + /* nattnums/attnums are populated below with attref_varnos */ + } + else + { + entry->njoinrels = 0; + entry->joinrels = NULL; + } + + /* + * For join stats, extract per-column varnos from the plain-column Var + * nodes in entry->exprs (in declared order). Each Var's varno tells + * which relation it belongs to (1=anchor, 2=first joined, etc.) + */ + if (entry->njoinrels > 0) + { + ListCell *elc; + int idx = 0; + + entry->nattnums = bms_num_members(entry->columns); + + entry->attnums = (int16 *) palloc(entry->nattnums * sizeof(int16)); + entry->attref_varnos = (int16 *) palloc(entry->nattnums * sizeof(int16)); + foreach(elc, entry->exprs) + { + Node *node = (Node *) lfirst(elc); + Var *var; + + if (!bms_is_member(foreach_current_index(elc), entry->columns)) + continue; + + var = (Var *) node; + entry->attnums[idx] = var->varattno; + entry->attref_varnos[idx] = var->varno; + idx++; + } + } + else + entry->attref_varnos = NULL; + + datum = SysCacheGetAttr(STATEXTOID, htup, + Anum_pg_statistic_ext_stxjoinconds, &isnull); + if (!isnull) + { + char *str = TextDatumGetCString(datum); + + entry->joinconds = (List *) stringToNode(str); + pfree(str); + } + else + entry->joinconds = NIL; + result = lappend(result, entry); } @@ -1051,7 +1320,7 @@ multi_sort_init(int ndims) { MultiSortSupport mss; - Assert(ndims >= 2); + Assert(ndims >= 1); /* join MCV stats may have a single column */ mss = (MultiSortSupport) palloc0(offsetof(MultiSortSupportData, ssup) + sizeof(SortSupportData) * ndims); @@ -1439,6 +1708,14 @@ choose_best_statistics(List *stats, char requiredkind, bool inh, if (info->inherit != inh) continue; + /* + * Skip join statistics. These describe the distribution over the + * join result, not the base table, so they must not be used to + * estimate single-table clauses. + */ + if (info->joinrels != NIL) + continue; + /* * Collect attributes and expressions in remaining (unestimated) * clauses fully covered by this statistic object. @@ -1534,7 +1811,7 @@ choose_best_statistics(List *stats, char requiredkind, bool inh, * Returns false if there is something we definitively can't handle. * On true return, we can proceed to match the *exprs against statistics. */ -static bool +bool statext_is_compatible_clause_internal(PlannerInfo *root, Node *clause, Index relid, Bitmapset **attnums, List **exprs, bool *leakproof) diff --git a/src/backend/statistics/extended_stats_funcs.c b/src/backend/statistics/extended_stats_funcs.c index 3f010533e25..58e8fe23c9f 100644 --- a/src/backend/statistics/extended_stats_funcs.c +++ b/src/backend/statistics/extended_stats_funcs.c @@ -581,6 +581,23 @@ extended_statistics_update(FunctionCallInfo fcinfo) if (has.mcv || has.expressions) { int idx; + Datum joinrels_datum; + bool joinrels_isnull; + oidvector *joinrels = NULL; + + /* + * The columns of join statistics can belong to any of the joined + * tables: varno N refers to stxjoinrels[N-1]. stxjoinrels is NULL + * for single-table statistics, whose columns all belong to relid. + * The joined tables need no lock: changing the type of a column used + * here makes ALTER TABLE drop and re-create these statistics, and + * dropping them locks relid. + */ + joinrels_datum = SysCacheGetAttr(STATEXTOID, tup, + Anum_pg_statistic_ext_stxjoinrels, + &joinrels_isnull); + if (!joinrels_isnull) + joinrels = (oidvector *) DatumGetPointer(joinrels_datum); atttypids = palloc0_array(Oid, numattrs); atttypmods = palloc0_array(int32, numattrs); @@ -598,12 +615,24 @@ extended_statistics_update(FunctionCallInfo fcinfo) if (bms_is_member(foreach_current_index(lc), columns)) { - AttrNumber attnum = ((Var *) node)->varattno; - HeapTuple atup = SearchSysCache2(ATTNUM, - ObjectIdGetDatum(relid), - Int16GetDatum(attnum)); + Var *var = (Var *) node; + AttrNumber attnum = var->varattno; + Oid colrelid = relid; + HeapTuple atup; Form_pg_attribute attr; + if (joinrels != NULL) + { + if (var->varno < 1 || var->varno > joinrels->dim1) + elog(ERROR, "stxexprs references invalid varno %d", + var->varno); + colrelid = joinrels->values[var->varno - 1]; + } + + atup = SearchSysCache2(ATTNUM, + ObjectIdGetDatum(colrelid), + Int16GetDatum(attnum)); + /* Attribute not found */ if (!HeapTupleIsValid(atup)) elog(ERROR, "stxexprs references nonexistent attnum %d", attnum); diff --git a/src/backend/statistics/join_mcv.c b/src/backend/statistics/join_mcv.c new file mode 100644 index 00000000000..15885ad1a22 --- /dev/null +++ b/src/backend/statistics/join_mcv.c @@ -0,0 +1,1469 @@ +/*------------------------------------------------------------------------- + * + * join_mcv.c + * POSTGRES join MCV lists + * + * Join MCV statistics are MCV lists over the rows of a join. The first + * table of the definition, the anchor, owns the statistics object, and + * ANALYZE of the anchor samples the join by looking up the rows of the + * other table through an index on its join column. + * + * Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * IDENTIFICATION + * src/backend/statistics/join_mcv.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#include "access/amapi.h" +#include "access/genam.h" +#include "access/heapam.h" +#include "access/htup_details.h" +#include "access/sysattr.h" +#include "access/table.h" +#include "access/tableam.h" +#include "catalog/pg_index.h" +#include "catalog/pg_statistic_ext.h" +#include "commands/vacuum.h" +#include "executor/tuptable.h" +#include "fmgr.h" +#include "nodes/nodeFuncs.h" +#include "optimizer/joininfo.h" +#include "optimizer/optimizer.h" +#include "optimizer/pathnode.h" +#include "optimizer/paths.h" +#include "rewrite/rewriteManip.h" +#include "statistics/extended_stats_internal.h" +#include "statistics/statistics.h" +#include "utils/datum.h" +#include "utils/lsyscache.h" +#include "utils/relcache.h" +#include "utils/sampling.h" +#include "utils/selfuncs.h" +#include "utils/snapmgr.h" +#include "utils/syscache.h" +#include "utils/typcache.h" + + +/* + * One filter clause that join statistics can estimate. + */ +typedef struct JoinMCVFilter +{ + RestrictInfo *rinfo; /* the filter clause */ + RelOptInfo *rel; /* the relation it filters */ + Expr *stat_clause; /* the clause with the stat's varnos */ +} JoinMCVFilter; + +/* + * One relation of a join statistics object, as the query joins it. + */ +typedef struct JoinStatRel +{ + RelOptInfo *rel; /* the relation in the query */ + Index stat_varno; /* its varno in the stat's definition */ + AttrNumber joinkey; /* its column in the stat's join condition */ +} JoinStatRel; + +/* + * Where sample_index() takes a column of the join sample from. + */ +typedef struct JoinSampleColumn +{ + bool anchor; /* is it a column of the anchor? */ + AttrNumber attnum; /* its attnum in its table */ + int16 typlen; + bool typbyval; +} JoinSampleColumn; + +/* + * Entries in the simplehash hash table used by sample_index() to group the + * anchor sample tuples by join value. + */ +typedef struct JoinValueGroup +{ + Datum value; /* the join value */ + List *rows; /* indexes of the sample tuples having it */ + uint32 hash; /* hash code for the Datum */ + char status; /* status code used by simplehash.h */ +} JoinValueGroup; + +/* forward reference */ +typedef struct JoinValueTable_hash JoinValueTable_hash; + +static uint32 hash_join_value(JoinValueTable_hash *tab, Datum key); +static bool join_values_equal(JoinValueTable_hash *tab, Datum key0, + Datum key1); + +/* Define support routines for the join value hash table */ +#define SH_PREFIX JoinValueTable +#define SH_ELEMENT_TYPE JoinValueGroup +#define SH_KEY_TYPE Datum +#define SH_KEY value +#define SH_HASH_KEY(tab,key) hash_join_value(tab, key) +#define SH_EQUAL(tab,key0,key1) join_values_equal(tab, key0, key1) +#define SH_SCOPE static inline +#define SH_STORE_HASH +#define SH_GET_HASH(tab,ent) (ent)->hash +#define SH_DEFINE +#define SH_DECLARE +#include "lib/simplehash.h" + + +/* + * find_index_for_operator + * Find an index that supports lookup on the given column using + * the specified operator. + * + * Returns InvalidOid if there is none. A unique index is preferred, then + * one with fewer key columns. + */ +Oid +find_index_for_operator(Relation rel, AttrNumber attnum, Oid eq_op) +{ + List *indexlist; + ListCell *lc; + Oid best_idx = InvalidOid; + bool best_is_unique = false; + int best_nattnums = INT_MAX; + + indexlist = RelationGetIndexList(rel); + + foreach(lc, indexlist) + { + Oid indexoid = lfirst_oid(lc); + Relation idxrel; + Form_pg_index idxform; + Oid opfamily; + + idxrel = index_open(indexoid, AccessShareLock); + idxform = idxrel->rd_index; + + /* + * A partial index would bias the sample, as it has only the rows + * matching its predicate. The lookup needs the column to lead the + * index, and an AM with amgettuple, which BRIN lacks. + */ + if (idxform->indisvalid && idxform->indisready && + idxform->indislive && + heap_attisnull(idxrel->rd_indextuple, Anum_pg_index_indpred, NULL) && + idxform->indnkeyatts >= 1 && + idxform->indkey.values[0] == attnum && + idxrel->rd_indam->amgettuple != NULL) + { + opfamily = idxrel->rd_opfamily[0]; + if (op_in_opfamily(eq_op, opfamily)) + { + if (!OidIsValid(best_idx) || + (!best_is_unique && idxform->indisunique) || + (idxform->indisunique == best_is_unique && + idxform->indnkeyatts < best_nattnums)) + { + best_idx = indexoid; + best_is_unique = idxform->indisunique; + best_nattnums = idxform->indnkeyatts; + } + } + } + + index_close(idxrel, AccessShareLock); + } + + list_free(indexlist); + return best_idx; +} + +/* + * join_stats_cond_vars + * Get the two columns compared by a join statistics condition. + * + * The parser wraps a column in a RelabelType when it compares it as a + * binary-compatible type, such as a varchar column compared as text. Look + * through any such relabeling, as the planner does when it examines a + * variable. + * + * Returns false if cond is not an operator with two column operands. + */ +bool +join_stats_cond_vars(Node *cond, Var **lvar, Var **rvar) +{ + OpExpr *op; + Node *left; + Node *right; + + if (!IsA(cond, OpExpr)) + return false; + op = (OpExpr *) cond; + if (list_length(op->args) != 2) + return false; + + left = linitial(op->args); + right = lsecond(op->args); + while (IsA(left, RelabelType)) + left = (Node *) ((RelabelType *) left)->arg; + while (IsA(right, RelabelType)) + right = (Node *) ((RelabelType *) right)->arg; + if (!IsA(left, Var) || !IsA(right, Var)) + return false; + + *lvar = (Var *) left; + *rvar = (Var *) right; + return true; +} + +/* + * join_stats_cond_sides + * Get the columns of a stored join statistics condition: the one of the + * table sampled first, and the one of the table probed for its rows. + * + * The tables are numbered so that the probed table of a condition has the + * higher varno. *commuted is set if the probed column is the left operand. + */ +void +join_stats_cond_sides(OpExpr *cond, Var **sampled, Var **probed, + bool *commuted) +{ + Var *lvar; + Var *rvar; + + if (!join_stats_cond_vars((Node *) cond, &lvar, &rvar)) + elog(ERROR, "unexpected join statistics condition"); + Assert(lvar->varno != rvar->varno); + + *commuted = lvar->varno > rvar->varno; + *sampled = *commuted ? rvar : lvar; + *probed = *commuted ? lvar : rvar; +} + +/* + * Support functions for the JoinValueTable hash table, which groups join + * values by their bytes. private_data is the join column's CompactAttribute. + */ +static uint32 +hash_join_value(JoinValueTable_hash *tab, Datum key) +{ + CompactAttribute *attr = (CompactAttribute *) tab->private_data; + + return datum_image_hash(key, attr->attbyval, attr->attlen); +} + +static bool +join_values_equal(JoinValueTable_hash *tab, Datum key0, Datum key1) +{ + CompactAttribute *attr = (CompactAttribute *) tab->private_data; + + return datum_image_eq(key0, key1, attr->attbyval, attr->attlen); +} + +/* + * store_join_row + * Store a pair of the join sample built by sample_index() as its row k. + * + * The pair is the anchor sample tuple anchor_tuple and the probed row in + * slot. Values already stored in row k are freed first, as the reservoir + * replaces rows. + */ +static void +store_join_row(StatsBuildData *data, int k, JoinSampleColumn *cols, + HeapTuple anchor_tuple, TupleDesc anchor_desc, + TupleTableSlot *slot) +{ + int f; + + for (f = 0; f < data->nattnums; f++) + { + Datum value; + bool isnull; + + if (k < data->numrows && !data->nulls[f][k] && !cols[f].typbyval) + pfree(DatumGetPointer(data->values[f][k])); + + if (cols[f].anchor) + value = heap_getattr(anchor_tuple, cols[f].attnum, anchor_desc, + &isnull); + else + value = slot_getattr(slot, cols[f].attnum, &isnull); + + if (!isnull && !cols[f].typbyval) + value = datumCopy(value, false, cols[f].typlen); + + data->values[f][k] = value; + data->nulls[f][k] = isnull; + } +} + +/* + * sample_index -- Build a random sample of the join result using an index. + * + * Given sample tuples S from the anchor table and an index I on the other + * table's join column, this produces a sample of the join of S with the other + * table, as Algorithm 1 from Leis et al., CIDR 2017, Section 3.1 does: if the + * join has N rows and the caller asks for at most n, it is a random sample of + * min(N, n) of them, drawn without replacement. + * + * Many tuples of S may have the same join value, and they all join the same + * rows of the other table. So rather than probing I once per tuple of S, we + * group the tuples of S by join value and probe I once per group, reading + * each matching row only once. Tuples whose join value is NULL join nothing + * and are skipped. + * + * The tuples are grouped by the bytes of their join value (datum_image_eq), + * not by an equality operator, so grouping can never disagree with the + * index's own equality. Values that are equal but stored differently, such + * as numeric 1.0 and 1.00, form separate groups, which costs an extra probe + * but does not change the sample. + * + * The join is sampled in one pass with reservoir sampling, as + * acquire_sample_rows() samples a table. A row read from the other table + * pairs with every tuple of its group, and a skip drawn from Vitter's + * Algorithm Z passes over all those pairs in one step. So the work grows + * with the rows read from the other table, not with the size of the join. + * + * Returns the sample, with the ncols columns cols in order, or NULL if no + * tuple of S joins. *result_joinrows is set to the number of join rows the + * tuples of S make, of which the sample is drawn. + */ +static StatsBuildData * +sample_index(HeapTuple *sample_tuples, int nsample, TupleDesc sample_desc, + AttrNumber joinkey_attnum, Oid joinkey_type, Oid join_op, + Relation heap_rel, Relation idx_rel, int n, + JoinSampleColumn *cols, int ncols, double *result_joinrows) +{ + IndexScanDesc iscan; + ScanKeyData skey[1]; + TupleTableSlot *slot; + int idx_strategy; + int i; + + /* the tuples of S grouped by join value */ + JoinValueTable_hash *groups; + JoinValueTable_iterator iter; + JoinValueGroup *group; + + /* reservoir sampling of the join rows */ + ReservoirStateData rstate; + double samplerows = 0; /* join rows seen so far */ + double rowstoskip = -1; /* -1 means not set yet */ + + StatsBuildData *result_data; + + Oid scan_eq_op; + RegProcedure eq_proc; + CompactAttribute *joinkey_attr; + + /* + * The scan key compares the index column with the anchor's join value as + * the type the join condition compares it as, joinkey_type, which differs + * from the column's own type under a binary-compatible relabeling. + */ + idx_strategy = get_op_opfamily_strategy(join_op, idx_rel->rd_opfamily[0]); + scan_eq_op = get_opfamily_member(idx_rel->rd_opfamily[0], + idx_rel->rd_opcintype[0], + joinkey_type, + idx_strategy); + if (!OidIsValid(scan_eq_op)) + elog(ERROR, "could not find equality operator for types %u and %u in opfamily %u", + idx_rel->rd_opcintype[0], joinkey_type, idx_rel->rd_opfamily[0]); + eq_proc = get_opcode(scan_eq_op); + + joinkey_attr = TupleDescCompactAttr(sample_desc, joinkey_attnum - 1); + groups = JoinValueTable_create(CurrentMemoryContext, nsample, + joinkey_attr); + + for (i = 0; i < nsample; i++) + { + Datum joinkey_value; + bool joinkey_isnull; + bool found; + + joinkey_value = heap_getattr(sample_tuples[i], joinkey_attnum, sample_desc, &joinkey_isnull); + if (joinkey_isnull) + continue; + + group = JoinValueTable_insert(groups, joinkey_value, &found); + if (!found) + { + /* + * Detoast once here rather than in every index comparison, as + * ExecIndexEvalRuntimeKeys() does. The bytes, and so the group, + * stay the same. + */ + if (joinkey_attr->attlen == -1) + group->value = PointerGetDatum(PG_DETOAST_DATUM(joinkey_value)); + group->rows = NIL; + } + group->rows = lappend_int(group->rows, i); + } + + /* Done if every join value is NULL */ + if (groups->members == 0) + return NULL; + + /* ANALYZE runs with an active snapshot, which the index scan uses */ + Assert(ActiveSnapshotSet()); + + slot = MakeSingleTupleTableSlot(RelationGetDescr(heap_rel), + &TTSOpsBufferHeapTuple); + iscan = index_beginscan(heap_rel, idx_rel, false, GetActiveSnapshot(), + NULL, 1, 0, 0); + + /* I.lookup(t): only the argument of the scan key changes per group */ + ScanKeyEntryInitialize(&skey[0], 0, 1, idx_strategy, joinkey_type, + idx_rel->rd_indcollation[0], + eq_proc, (Datum) 0); + + /* The reservoir: at most n join rows */ + result_data = palloc(sizeof(StatsBuildData)); + result_data->nattnums = ncols; + result_data->numrows = 0; + result_data->attnums = palloc_array(AttrNumber, ncols); + result_data->stats = palloc0_array(VacAttrStats *, ncols); + result_data->values = palloc_array(Datum *, ncols); + result_data->nulls = palloc_array(bool *, ncols); + + for (i = 0; i < ncols; i++) + { + /* + * The columns come from two tables, so their attnums can collide. + * Number them by their position instead. + */ + result_data->attnums[i] = i + 1; + result_data->values[i] = palloc_array(Datum, n); + result_data->nulls[i] = palloc_array(bool, n); + } + + reservoir_init_selection_state(&rstate, n); + + JoinValueTable_start_iterate(groups, &iter); + while ((group = JoinValueTable_iterate(groups, &iter)) != NULL) + { + int nrows = list_length(group->rows); + + /* I.lookup(t), for every tuple t of the group */ + skey[0].sk_argument = group->value; + index_rescan(iscan, skey, 1, NULL, 0); + + while (table_index_getnext_slot(iscan, ForwardScanDirection, slot)) + { + int j; + + vacuum_delay_point(true); + + /* a lossy index, such as hash, returns rows to recheck */ + if (iscan->xs_recheck) + { + Datum val; + bool isnull; + + val = slot_getattr(slot, + idx_rel->rd_index->indkey.values[0], + &isnull); + if (isnull || + !DatumGetBool(FunctionCall2Coll(&skey[0].sk_func, + skey[0].sk_collation, + val, + skey[0].sk_argument))) + continue; + } + + /* + * The row pairs with each of the nrows tuples of the group. The + * pairs fill the reservoir first. After that, each pair the + * sampler selects replaces a random one in the reservoir, and the + * pairs in between are skipped without being looked at. + */ + j = 0; + while (j < nrows) + { + int k; + + if (result_data->numrows < n) + { + store_join_row(result_data, result_data->numrows, cols, + sample_tuples[list_nth_int(group->rows, j)], + sample_desc, slot); + result_data->numrows++; + } + else + { + /* + * t in Vitter's paper is the number of records already + * processed, here samplerows, all the pairs before the + * j-th pair of this row. + */ + if (rowstoskip < 0) + rowstoskip = reservoir_get_next_S(&rstate, samplerows, n); + + /* Skip the rest of this row's pairs? */ + if (rowstoskip >= nrows - j) + { + rowstoskip -= nrows - j; + samplerows += nrows - j; + break; + } + + j += (int) rowstoskip; + samplerows += rowstoskip; + rowstoskip = -1; + + k = (int) (n * sampler_random_fract(&rstate.randstate)); + Assert(k >= 0 && k < n); + store_join_row(result_data, k, cols, + sample_tuples[list_nth_int(group->rows, j)], + sample_desc, slot); + } + + j++; + samplerows += 1; + } + } + + list_free(group->rows); + } + + JoinValueTable_destroy(groups); + index_endscan(iscan); + ExecDropSingleTupleTableSlot(slot); + + if (result_data->numrows == 0) + return NULL; + + *result_joinrows = samplerows; + return result_data; +} + +/* + * statext_join_mcv_build + * Build join MCV statistics using index-based join sampling. + * + * Samples the join of the anchor's ANALYZE sample rows with the other table, + * looking rows up through index_oid, and builds an MCV list from the sample. + * + * joinrel_oids and joinconds are the stat's tables, the anchor first, and + * its join conditions. stxkeys are the stat's columns, keyrefs their + * varnos. + */ +MCVList * +statext_join_mcv_build(Relation anchor_rel, + Oid *joinrel_oids, + int njoinrels, + List *joinconds, + Oid index_oid, + int16 *stxkeys, + int16 *keyrefs, + int nkeys, + int stattarget, + int numrows, HeapTuple *rows, + double totalrows, + VacAttrStats ***result_stats) +{ + OpExpr *cond; + Var *sampled_var; + Var *probed_var; + bool commuted; + TupleDesc anchor_desc = RelationGetDescr(anchor_rel); + Relation other_rel; + TupleDesc other_desc; + Relation idx_rel; + JoinSampleColumn *cols; + StatsBuildData *data; + double joinrows; + MCVList *mcv; + int i; + + /* the caller skips the shapes that cannot be built yet */ + Assert(njoinrels == 2 && list_length(joinconds) == 1); + + cond = linitial_node(OpExpr, joinconds); + join_stats_cond_sides(cond, &sampled_var, &probed_var, &commuted); + + other_rel = table_open(joinrel_oids[1], AccessShareLock); + other_desc = RelationGetDescr(other_rel); + idx_rel = index_open(index_oid, AccessShareLock); + + /* keyrefs[i] is 1 for a column of the anchor, 2 for the other table */ + cols = palloc_array(JoinSampleColumn, nkeys); + for (i = 0; i < nkeys; i++) + { + Form_pg_attribute attr; + + cols[i].anchor = (keyrefs[i] == 1); + cols[i].attnum = stxkeys[i]; + attr = TupleDescAttr(cols[i].anchor ? anchor_desc : other_desc, + stxkeys[i] - 1); + cols[i].typlen = attr->attlen; + cols[i].typbyval = attr->attbyval; + } + + /* + * XXX ANALYZE samples 300 rows per unit of statistics target for + * single-table statistics; whether the join sample should match is open. + */ + data = sample_index(rows, numrows, anchor_desc, + sampled_var->varattno, + exprType(commuted ? lsecond(cond->args) : + linitial(cond->args)), + cond->opno, other_rel, idx_rel, stattarget * 100, + cols, nkeys, &joinrows); + + index_close(idx_rel, AccessShareLock); + + if (data == NULL) + { + table_close(other_rel, AccessShareLock); + return NULL; + } + + for (i = 0; i < nkeys; i++) + { + TupleDesc desc = cols[i].anchor ? anchor_desc : other_desc; + Form_pg_attribute attr = TupleDescAttr(desc, cols[i].attnum - 1); + VacAttrStats *s; + HeapTuple typtuple; + + s = palloc0(sizeof(VacAttrStats)); + s->tupDesc = desc; + s->tupattnum = i + 1; + s->attrtypid = attr->atttypid; + s->attrtypmod = attr->atttypmod; + s->attrcollid = attr->attcollation; + + typtuple = SearchSysCache1(TYPEOID, + ObjectIdGetDatum(attr->atttypid)); + if (!HeapTupleIsValid(typtuple)) + elog(ERROR, "cache lookup failed for type %u", attr->atttypid); + s->attrtype = (Form_pg_type) palloc(sizeof(FormData_pg_type)); + memcpy(s->attrtype, GETSTRUCT(typtuple), + sizeof(FormData_pg_type)); + ReleaseSysCache(typtuple); + data->stats[i] = s; + } + + /* + * statext_mcv_build() needs the number of rows the sample was drawn from, + * the size of the join. Estimate it as acquire_sample_rows() estimates a + * table's size, scaling up from the sampled anchor rows. + */ + mcv = statext_mcv_build(data, joinrows * totalrows / numrows, + stattarget); + + table_close(other_rel, AccessShareLock); + + *result_stats = data->stats; + return mcv; +} + +/* + * join_mcv_base_selectivity - + * Sum of the base frequencies of the matching MCV items. + * + * The base frequency of an item is the share of the join that would hold + * its values if its columns were unrelated to the join: the product, over + * the dimensions, of the standard estimate of "column = value" on the + * column's table. The single-table code reads base_frequency from the MCV + * list, the product of the per-column frequencies in the same sample. Here + * the sample is the join, while the shares the standard estimate assumes are + * those in the tables. + * + * rels are the nrels relations of the stat, as the query joins them, and + * matched[] marks the items that match the filters. + */ +static Selectivity +join_mcv_base_selectivity(PlannerInfo *root, + StatisticExtInfo *stat, + JoinStatRel *rels, + int nrels, + MCVList *mcvlist, + bool *matched) +{ + Selectivity basesel = 0.0; + VariableStatData *vardatas; + Var **vars; + Oid *eqops; + ListCell *lc; + int dim; + int i; + + vardatas = palloc_array(VariableStatData, mcvlist->ndimensions); + vars = palloc_array(Var *, mcvlist->ndimensions); + eqops = palloc_array(Oid, mcvlist->ndimensions); + + /* The dimensions of the MCV list are the stat's columns, in order */ + dim = 0; + foreach(lc, stat->exprs) + { + Var *var; + TypeCacheEntry *typentry; + + /* join statistics have no expressions */ + if (!bms_is_member(foreach_current_index(lc), stat->columns)) + continue; + + /* the stat's columns carry its varnos, and rels are in varno order */ + var = copyObject(lfirst_node(Var, lc)); + Assert(var->varno >= 1 && var->varno <= nrels && + rels[var->varno - 1].stat_varno == var->varno); + ChangeVarNodes((Node *) var, var->varno, + rels[var->varno - 1].rel->relid, 0); + + examine_variable(root, (Node *) var, 0, &vardatas[dim]); + vars[dim] = var; + + /* the MCV list is built with the type's btree opclass, which has = */ + typentry = lookup_type_cache(mcvlist->types[dim], TYPECACHE_EQ_OPR); + Assert(OidIsValid(typentry->eq_opr)); + eqops[dim] = typentry->eq_opr; + + dim++; + } + Assert(dim == mcvlist->ndimensions); + + for (i = 0; i < mcvlist->nitems; i++) + { + MCVItem *item = &mcvlist->items[i]; + Selectivity itemsel = 1.0; + + if (!matched[i]) + continue; + + for (dim = 0; dim < mcvlist->ndimensions; dim++) + { + if (item->isnull[dim]) + itemsel *= nulltestsel(root, IS_NULL, (Node *) vars[dim], 0, + JOIN_INNER, NULL); + else + itemsel *= var_eq_const(&vardatas[dim], eqops[dim], + vars[dim]->varcollid, + item->values[dim], false, true, false); + } + + basesel += itemsel; + } + + for (dim = 0; dim < mcvlist->ndimensions; dim++) + ReleaseVariableStats(vardatas[dim]); + pfree(vardatas); + pfree(vars); + pfree(eqops); + + return basesel; +} + +/* + * join_mcv_clauselist_selectivity - + * Estimate the share of the join that satisfies the filters. + * + * Returns the sum of the frequencies of the items of the stat's MCV list that + * match every filter in filters, a List of JoinMCVFilter. An item's + * frequency is a fraction of the sampled join, not of a table's rows. + * Dimensions no filter covers are left unconstrained, so with statistics on + * (category, color) a filter on color alone matches every item storing that + * color. + * + * rels are the nrels relations of the stat, as the query joins them. + * *basesel and *totalsel are set as mcv_clauselist_selectivity() sets them + * for the single-table code, for mcv_combine_selectivities(). + */ +static Selectivity +join_mcv_clauselist_selectivity(PlannerInfo *root, + StatisticExtInfo *stat, + JoinStatRel *rels, + int nrels, + List *filters, + Selectivity *basesel, + Selectivity *totalsel) +{ + MCVList *mcvlist; + Selectivity sel = 0.0; + List *clauses = NIL; + bool *matched; + int i; + + mcvlist = statext_mcv_load(stat->statOid, false); + + *totalsel = 0.0; + for (i = 0; i < mcvlist->nitems; i++) + *totalsel += mcvlist->items[i].frequency; + + /* the filters carry the stat's varnos, as the MCV list's columns do */ + foreach_ptr(JoinMCVFilter, filter, filters) + clauses = lappend(clauses, filter->stat_clause); + + matched = mcv_get_match_bitmap(root, clauses, stat->exprs, mcvlist, false); + + for (i = 0; i < mcvlist->nitems; i++) + if (matched[i]) + sel += mcvlist->items[i].frequency; + + *basesel = join_mcv_base_selectivity(root, stat, rels, nrels, mcvlist, + matched); + + return sel; +} + +/* + * filter_rows_selectable - + * Could the user read the rows that a filter's join MCV values come from? + * + * Those values come from the rows of the relations that join, so the user + * must be able to read every relation's join key and filter_attnums, the + * plain attnums of the filtered columns of filter_rel, with no RLS or + * security barrier quals on any of the relations. This is the join + * counterpart of the all_rows_selectable() check in + * statext_is_compatible_clause(). + */ +static bool +filter_rows_selectable(PlannerInfo *root, JoinStatRel *rels, int nrels, + JoinStatRel *filter_rel, Bitmapset *filter_attnums) +{ + int i; + + for (i = 0; i < nrels; i++) + { + Bitmapset *attnums; + bool selectable; + + attnums = bms_make_singleton(rels[i].joinkey - + FirstLowInvalidHeapAttributeNumber); + if (&rels[i] == filter_rel) + attnums = bms_join(attnums, + bms_offset_members(bms_copy(filter_attnums), + 0 - FirstLowInvalidHeapAttributeNumber)); + + selectable = all_rows_selectable(root, rels[i].rel->relid, attnums); + bms_free(attnums); + if (!selectable) + return false; + } + + return true; +} + +/* + * stat_covers_column - + * Is column attnum of the stat's relation stat_varno a column of the stat? + */ +static bool +stat_covers_column(StatisticExtInfo *stat, Index stat_varno, AttrNumber attnum) +{ + ListCell *lc; + + foreach(lc, stat->exprs) + { + Var *var = (Var *) lfirst(lc); + + /* join statistics have no expressions */ + if (!bms_is_member(foreach_current_index(lc), stat->columns)) + continue; + + if (var->varno == stat_varno && var->varattno == attnum) + return true; + } + + return false; +} + +/* + * match_filters_for_rel - + * Find the filters on one relation of the join that the stat can estimate. + * + * Appends to *filters a JoinMCVFilter for each restriction clause of + * statrel, one of the nrels relations in rels, that is on columns of the stat + * and that the single-table MCV code could estimate. Estimation calls the + * clause's operator on the stored values, so a leaky operator is used only if + * the user could read the rows those values come from. + */ +static void +match_filters_for_rel(PlannerInfo *root, + StatisticExtInfo *stat, + JoinStatRel *rels, + int nrels, + JoinStatRel *statrel, + List **filters) +{ + RelOptInfo *rel = statrel->rel; + Index stat_varno = statrel->stat_varno; + ListCell *lc; + + foreach(lc, rel->baserestrictinfo) + { + RestrictInfo *ri = lfirst_node(RestrictInfo, lc); + JoinMCVFilter *filter; + Bitmapset *attnums = NULL; + List *exprs = NIL; + bool leakproof = true; + bool covered = true; + int attnum = -1; + + if (!statext_is_compatible_clause_internal(root, (Node *) ri->clause, + rel->relid, &attnums, + &exprs, &leakproof)) + continue; + + /* join statistics have no expressions */ + if (exprs != NIL || bms_is_empty(attnums)) + continue; + + while ((attnum = bms_next_member(attnums, attnum)) >= 0) + { + if (!stat_covers_column(stat, stat_varno, attnum)) + { + covered = false; + break; + } + } + if (!covered) + continue; + + /* Skip a leaky operator unless the user could read the rows anyway */ + if (!leakproof && + !filter_rows_selectable(root, rels, nrels, statrel, attnums)) + continue; + + filter = palloc_object(JoinMCVFilter); + filter->rinfo = ri; + filter->rel = rel; + + /* the stat's columns carry its varnos, not the query's */ + filter->stat_clause = copyObject(ri->clause); + ChangeVarNodes((Node *) filter->stat_clause, rel->relid, + stat_varno, 0); + + *filters = lappend(*filters, filter); + } +} + +/* + * clause_is_join_condition - + * Does a clause of the query equate the columns of a stat's condition? + * + * The clause must be "Var op Var" between anchor_relids and other_relids on + * the condition's columns, with the condition's operator or its commutator. + * Nulling bits on the Vars are ignored, as for a foreign key column in + * match_foreign_keys_to_quals(). + */ +static bool +clause_is_join_condition(Expr *clause, + Relids anchor_relids, Relids other_relids, + AttrNumber anchor_key, AttrNumber other_key, + Oid opno) +{ + Var *lvar; + Var *rvar; + Oid clause_opno; + + if (!join_stats_cond_vars((Node *) clause, &lvar, &rvar)) + return false; + clause_opno = ((OpExpr *) clause)->opno; + + /* Orient the clause as (anchor, other) */ + if (bms_is_member(lvar->varno, other_relids)) + { + Var *tmp = lvar; + + lvar = rvar; + rvar = tmp; + clause_opno = get_commutator(clause_opno); + } + + return bms_is_member(lvar->varno, anchor_relids) && + bms_is_member(rvar->varno, other_relids) && + lvar->varattno == anchor_key && + rvar->varattno == other_key && + clause_opno == opno; +} + +/* + * ec_has_join_condition - + * Does an EquivalenceClass equate the two columns of a stat's condition? + * + * Both columns must be members of the class, as plain Vars matched by varno + * and attno (nulling bits are ignored), and the class's operator families + * must be those of the condition's operator, as + * match_eclasses_to_foreign_key_col() matches a foreign key column. + */ +static bool +ec_has_join_condition(EquivalenceClass *ec, + RelOptInfo *anchor_rel, RelOptInfo *other_rel, + AttrNumber anchor_key, AttrNumber other_key, + Oid opno) +{ + EquivalenceMemberIterator it; + EquivalenceMember *em; + Relids join_relids; + bool anchor_found = false; + bool other_found = false; + + join_relids = bms_union(anchor_rel->relids, other_rel->relids); + setup_eclass_member_iterator(&it, ec, join_relids); + while ((em = eclass_member_iterator_next(&it)) != NULL) + { + Var *var = (Var *) em->em_expr; + + while (var && IsA(var, RelabelType)) + var = (Var *) ((RelabelType *) var)->arg; + if (!(var && IsA(var, Var))) + continue; + + if (bms_is_member(var->varno, anchor_rel->relids) && + var->varattno == anchor_key) + anchor_found = true; + else if (bms_is_member(var->varno, other_rel->relids) && + var->varattno == other_key) + other_found = true; + } + bms_free(join_relids); + + if (!anchor_found || !other_found) + return false; + + return equal(get_mergejoin_opfamilies(opno), ec->ec_opfamilies); +} + +/* + * clauses_have_join_condition - + * Do the join clauses estimated at a join step enforce a stat's join + * condition between anchor_rel and other_rel? + * + * A clause enforces the condition if it is the condition itself, or comes + * from an EquivalenceClass equating the two columns. For a join the planner + * generates one clause per class, between whichever two members it picks, so + * the class rather than the clause shows the condition holds at this step. + */ +static bool +clauses_have_join_condition(List *clauses, + RelOptInfo *anchor_rel, RelOptInfo *other_rel, + AttrNumber anchor_key, AttrNumber other_key, + Oid opno) +{ + ListCell *lc; + + foreach(lc, clauses) + { + RestrictInfo *rinfo; + EquivalenceClass *ec; + + if (!IsA(lfirst(lc), RestrictInfo)) + continue; + rinfo = (RestrictInfo *) lfirst(lc); + + if (clause_is_join_condition(rinfo->clause, + anchor_rel->relids, other_rel->relids, + anchor_key, other_key, opno)) + return true; + + /* + * A generated clause records its class in parent_ec; a source clause + * of a class, reused as it is, has the class as both left_ec and + * right_ec. + */ + ec = rinfo->parent_ec; + if (ec == NULL && rinfo->left_ec != NULL && + rinfo->left_ec == rinfo->right_ec) + ec = rinfo->left_ec; + if (ec != NULL && + ec_has_join_condition(ec, anchor_rel, other_rel, + anchor_key, other_key, opno)) + return true; + } + + return false; +} + +/* + * stat_describes_join - + * Does a stat describe the join of anchor_rel and other_rel being + * estimated? + * + * The stat's condition must be among clauses, the join clauses estimated at + * this step, and the query must have no other join condition between the two + * relations. Those are counted from the planner's data rather than from + * clauses, since a caller may estimate only some of them, such as the + * clauses of a foreign key. Requiring the condition among the step's + * clauses is what applies a stat exactly once: a clause pushed down into a + * parameterized path is estimated there, and is absent from the join above + * it. + * + * Sets *anchor_key and *other_key to the columns of the stat's condition. + * + * XXX Several conditions would need each of the stat's conditions matched to + * a distinct join condition of the query, and composite key lookups in + * ANALYZE. + */ +static bool +stat_describes_join(PlannerInfo *root, + StatisticExtInfo *stat, + RelOptInfo *anchor_rel, + RelOptInfo *other_rel, + List *clauses, + AttrNumber *anchor_key, + AttrNumber *other_key) +{ + OpExpr *cond; + Var *anchor_var; + Var *other_var; + bool commuted; + Oid opno; + + if (list_length(stat->joinconds) != 1) + return false; + + cond = linitial_node(OpExpr, stat->joinconds); + join_stats_cond_sides(cond, &anchor_var, &other_var, &commuted); + Assert(anchor_var->varno == 1 && other_var->varno == 2); + + /* the operator for the (anchor, other) order */ + opno = commuted ? get_commutator(cond->opno) : cond->opno; + if (!OidIsValid(opno)) + return false; + + *anchor_key = anchor_var->varattno; + *other_key = other_var->varattno; + + if (count_relevant_joinclauses(root, anchor_rel, other_rel) != 1) + return false; + + return clauses_have_join_condition(clauses, anchor_rel, other_rel, + *anchor_key, *other_key, opno); +} + +/* + * join_mcv_combine_selectivities + * Combine a join clause's selectivity with the join MCV selectivity + * estimates of the filters. + * + * join_sel is an estimate of the join clause alone: the standard one for an + * ordinary join, or the foreign key's in get_foreign_key_join_selectivity(). + * The join MCV statistics do not replace it: their frequencies are fractions + * of the joined rows, so what they estimate is how often the covered filters + * hold among the rows that join. + * + * covered_join_sel is that estimate, the selectivity of the covered filters + * over the joined rows, P(covered_filters | join). covered_rels_sel is the + * product, over the joined relations, of the selectivity of each relation's + * covered filters over that relation. The planner computes: + * + * nrows = rows(rel_1) * rows(rel_2) * ... * sel(clause) + * + * where each relation's rows already reflect its base restrictions, the + * covered filters among them. Of the rows of the join without filters, + * covered_join_sel satisfy the covered filters. Dividing out the share of + * the relations' rows the covered filters account for gives: + * + * sel = join_sel * covered_join_sel / covered_rels_sel + * + * The quotient is 1 if the filters are unrelated to the join. The result is + * not clamped, as the caller applies the corrections of all its pairs and + * clamps once. + */ +static Selectivity +join_mcv_combine_selectivities(Selectivity join_sel, + Selectivity covered_join_sel, + Selectivity covered_rels_sel) +{ + Assert(covered_rels_sel > 0); + + return join_sel * covered_join_sel / covered_rels_sel; +} + +/* + * join_mcv_pair_selectivities + * Estimate the selectivity of the filters covered by join MCV + * statistics, within the join and in the tables. + * + * Searches the statlists of a pair of base relations for a join stat that + * describes their join, given the join clauses being estimated. *joinsel is + * set to the share of the join that satisfies the filters the stat covers, + * and *relssel to the product, over the two relations, of the selectivity of + * each relation's covered filters. The caller combines the two with its + * estimate of the join clause using join_mcv_combine_selectivities(). + * Uncovered filters need nothing here, as they are already reflected in the + * relations' rows. + * + * Returns false if no join stat applies. *joinsel may be 0, when a stat + * storing every combination in the join has none matching the filters. + */ +static bool +join_mcv_pair_selectivities(PlannerInfo *root, + RelOptInfo *baserel1, + RelOptInfo *baserel2, + List *clauses, + Selectivity *joinsel, + Selectivity *relssel) +{ + RelOptInfo *rels[2]; + int r; + Selectivity best_join_sel = 0.0; + Selectivity best_rels_sel = 0.0; + int best_ncovered = 0; + + if (baserel1->rtekind != RTE_RELATION || baserel2->rtekind != RTE_RELATION) + return false; + + /* The floors on the covered selectivities below divide by tuples */ + if (baserel1->tuples <= 0 || baserel2->tuples <= 0) + return false; + + rels[0] = baserel1; + rels[1] = baserel2; + + /* + * A join stat is in the statlist of its anchor, so try each relation as + * the anchor. A stat covering all the filters is used at once, and + * otherwise the one covering the most. + */ + for (r = 0; r < 2; r++) + { + RelOptInfo *anchor_rel = rels[r]; + RelOptInfo *other_rel = rels[1 - r]; + Oid other_reloid = root->simple_rte_array[other_rel->relid]->relid; + ListCell *lc; + int nfilters; /* total baserestrictinfo clauses on both rels */ + + nfilters = list_length(other_rel->baserestrictinfo) + + list_length(anchor_rel->baserestrictinfo); + + foreach(lc, anchor_rel->statlist) + { + StatisticExtInfo *stat = (StatisticExtInfo *) lfirst(lc); + JoinStatRel statrels[2]; + List *filters; + Selectivity covered_join_sel; + Selectivity covered_rels_sel; + Selectivity mcv_basesel; + Selectivity mcv_totalsel; + AttrNumber anchor_key; + AttrNumber other_key; + int ncovered; /* number of filters the stat covers */ + int i; + + /* Skip non-join and non-MCV stats */ + if (stat->joinrels == NIL || stat->kind != STATS_EXT_MCV) + continue; + + /* Check that this stat joins to the other relation */ + if (!list_member_oid(stat->joinrels, other_reloid)) + continue; + + if (!stat_describes_join(root, stat, anchor_rel, other_rel, clauses, + &anchor_key, &other_key)) + continue; + + /* the stat's definition numbers the anchor 1, the other table 2 */ + statrels[0].rel = anchor_rel; + statrels[0].stat_varno = 1; + statrels[0].joinkey = anchor_key; + statrels[1].rel = other_rel; + statrels[1].stat_varno = 2; + statrels[1].joinkey = other_key; + + /* + * Which filters does it cover? Without at least one, the stat + * cannot improve on the standard join selectivity estimate. + */ + filters = NIL; + for (i = 0; i < lengthof(statrels); i++) + match_filters_for_rel(root, stat, statrels, lengthof(statrels), + &statrels[i], &filters); + if (filters == NIL) + continue; + ncovered = list_length(filters); + + /* + * The selectivity of each relation's covered filters, through + * clauselist_selectivity() so that single-table extended + * statistics are used, as they were for the relation's rows. It + * is at least one row, as set_baserel_size_estimates() clamps the + * rows to at least one. When the covered filters are all of the + * relation's restrictions, this is that clamp exactly. + */ + covered_rels_sel = 1.0; + for (i = 0; i < lengthof(statrels); i++) + { + List *rinfos = NIL; + Selectivity sel = 1.0; + + foreach_ptr(JoinMCVFilter, filter, filters) + { + if (filter->rel == statrels[i].rel) + rinfos = lappend(rinfos, filter->rinfo); + } + if (rinfos != NIL) + sel = clauselist_selectivity(root, rinfos, 0, JOIN_INNER, + NULL); + covered_rels_sel *= Max(sel, 1.0 / statrels[i].rel->tuples); + } + + /* + * Combine the share of the join the matching items measure with + * the standard estimate for the rest of the join, as + * mcv_combine_selectivities() does for the single-table code. The + * standard estimate here is covered_rels_sel, the share of the + * join the filters select if they are unrelated to it. + */ + covered_join_sel = join_mcv_clauselist_selectivity(root, stat, + statrels, + lengthof(statrels), + filters, + &mcv_basesel, + &mcv_totalsel); + covered_join_sel = mcv_combine_selectivities(covered_rels_sel, + covered_join_sel, + mcv_basesel, + mcv_totalsel); + + /* Full coverage: return immediately */ + if (ncovered >= nfilters) + { + *joinsel = covered_join_sel; + *relssel = covered_rels_sel; + return true; + } + + /* Partial coverage: save if better than previous best */ + if (ncovered > best_ncovered) + { + /* + * With a relation's rows at the one-row floor, the uncovered + * filters are seen only through that clamped value, so a + * result that raises the estimate would divide out the + * covered filters alone and inflate it. A result that lowers + * the estimate is kept: adding the uncovered filters cannot + * make the covered ones more common in the join. + */ + if (covered_join_sel > covered_rels_sel && + (other_rel->rows <= 1.0 || anchor_rel->rows <= 1.0)) + continue; + + best_join_sel = covered_join_sel; + best_rels_sel = covered_rels_sel; + best_ncovered = ncovered; + } + } + } + + if (best_ncovered > 0) + { + *joinsel = best_join_sel; + *relssel = best_rels_sel; + return true; + } + + return false; +} + +/* + * join_mcv_join_selectivity + * Correct the selectivity of a join step with join MCV statistics. + * + * outer_relids and inner_relids are the relations joined at this step, + * clauses the join clauses estimated at it (all of them, including any a + * foreign key accounts for), and join_sel the planner's estimate of those + * clauses: the product of their standard selectivities and of any foreign + * key's, or their estimate as restrictions for a parameterized scan. For + * each pair of base relations across the two sides whose join a stat + * describes, join_sel is corrected for the filters the stat covers, with + * join_mcv_combine_selectivities(). Returns the corrected selectivity. + * + * A stat is applied at the step whose clauses enforce its join condition, + * so its correction enters the join's estimate once, whatever the join order + * and however the condition appears: as a clause of the step, removed by a + * foreign key, generated by an EquivalenceClass, or pushed down into a + * parameterized scan, which is then corrected instead of the join above it. + * The corrections of the pairs are multiplied and the result clamped once. + */ +Selectivity +join_mcv_join_selectivity(PlannerInfo *root, + Relids outer_relids, + Relids inner_relids, + List *clauses, + Selectivity join_sel) +{ + int outer_relid = -1; + + /* Nothing to search for if no relation of the query has join stats */ + if (!root->hasJoinStats) + return join_sel; + + while ((outer_relid = bms_next_member(outer_relids, outer_relid)) >= 0) + { + RelOptInfo *outer_rel = find_base_rel_ignore_join(root, outer_relid); + int inner_relid = -1; + + if (outer_rel == NULL) + continue; + + while ((inner_relid = bms_next_member(inner_relids, inner_relid)) >= 0) + { + RelOptInfo *inner_rel = find_base_rel_ignore_join(root, inner_relid); + Selectivity covered_join_sel; + Selectivity covered_rels_sel; + + if (inner_rel == NULL) + continue; + + if (join_mcv_pair_selectivities(root, outer_rel, inner_rel, + clauses, + &covered_join_sel, + &covered_rels_sel)) + join_sel = join_mcv_combine_selectivities(join_sel, + covered_join_sel, + covered_rels_sel); + } + } + CLAMP_PROBABILITY(join_sel); + + return join_sel; +} + +/* + * join_mcv_restriction_selectivity + * Correct the selectivity of a join clause estimated as a restriction. + * + * A join clause is estimated as a restriction on varRelid when a scan of + * that relation applies it, parameterized by the other relations the clause + * mentions: get_parameterized_baserel_size() estimates the scan's rows that + * way, and the index cost estimators the quals of its indexes. The scan + * enforces the clause, so its join statistics apply here, to sel. The join + * above the scan does not see the clause and does not correct for it again. + * + * Not when the relation is the left-hand side of a semi join whose + * right-hand side supplies the parameters. That side is made unique before + * the join, so the scan's outer rows are not joined rows, of which the + * statistics' frequencies are fractions. + */ +Selectivity +join_mcv_restriction_selectivity(PlannerInfo *root, + RestrictInfo *rinfo, + int varRelid, + Selectivity sel) +{ + RelOptInfo *rel; + Relids outer_relids; + ListCell *lc; + + if (!root->hasJoinStats) + return sel; + + rel = find_base_rel_ignore_join(root, varRelid); + if (rel == NULL) + return sel; + + outer_relids = bms_del_member(bms_copy(rinfo->clause_relids), varRelid); + + foreach(lc, root->join_info_list) + { + SpecialJoinInfo *sjinfo = (SpecialJoinInfo *) lfirst(lc); + + if (sjinfo->jointype == JOIN_SEMI && + bms_is_member(varRelid, sjinfo->syn_lefthand) && + bms_overlap(outer_relids, sjinfo->syn_righthand)) + return sel; + } + + return join_mcv_join_selectivity(root, outer_relids, rel->relids, + list_make1(rinfo), sel); +} diff --git a/src/backend/statistics/mcv.c b/src/backend/statistics/mcv.c index e0e329e4918..35b2fc02b59 100644 --- a/src/backend/statistics/mcv.c +++ b/src/backend/statistics/mcv.c @@ -1544,7 +1544,12 @@ mcv_match_expression(Node *expr, List *exprs, Oid *collid) if (expr_is_var) { + /* + * Compare varno too: a join statistics object has columns of + * several relations, which may have the same attnum. + */ if (IsA(node, Var) && + ((Var *) node)->varno == ((Var *) expr)->varno && ((Var *) node)->varattno == ((Var *) expr)->varattno) return idx; } @@ -1581,7 +1586,7 @@ mcv_match_expression(Node *expr, List *exprs, Oid *collid) * & and |, which should be faster than min/max. The bitmaps are fairly * small, though (thanks to the cap on the MCV list size). */ -static bool * +bool * mcv_get_match_bitmap(PlannerInfo *root, List *clauses, List *exprs, MCVList *mcvlist, bool is_or) diff --git a/src/backend/statistics/meson.build b/src/backend/statistics/meson.build index 9a7bf55e301..450b5ef0c64 100644 --- a/src/backend/statistics/meson.build +++ b/src/backend/statistics/meson.build @@ -5,6 +5,7 @@ backend_sources += files( 'dependencies.c', 'extended_stats.c', 'extended_stats_funcs.c', + 'join_mcv.c', 'mcv.c', 'mvdistinct.c', 'relation_stats.c', diff --git a/src/backend/tcop/utility.c b/src/backend/tcop/utility.c index 1512d2df196..fe7a6ad8139 100644 --- a/src/backend/tcop/utility.c +++ b/src/backend/tcop/utility.c @@ -1880,45 +1880,58 @@ ProcessUtilitySlow(ParseState *pstate, { Oid relid; CreateStatsStmt *stmt = (CreateStatsStmt *) parsetree; - RangeVar *rel; + Node *fromNode; + List *relids = NIL; /* - * Examine the FROM clause. Currently, we only allow it - * to be a single simple table, but later we'll probably - * allow multiple tables and JOIN syntax. The grammar is - * already prepared for that, so we have to check here - * that what we got is what we can support. + * Examine the FROM clause. It must be a single item, + * either a simple table for single-table statistics or a + * JOIN for join statistics; the grammar accepts a list, + * so we have to check here that what we got is what we + * can support. */ if (list_length(stmt->relations) != 1) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("only a single relation is allowed in CREATE STATISTICS"))); + errmsg("only a single relation or JOIN is allowed in CREATE STATISTICS"))); - rel = (RangeVar *) linitial(stmt->relations); + fromNode = (Node *) linitial(stmt->relations); - if (!IsA(rel, RangeVar)) - ereport(ERROR, - (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), - errmsg("CREATE STATISTICS only supports relation names in the FROM clause"))); + if (IsA(fromNode, RangeVar)) + { + RangeVar *rel = (RangeVar *) fromNode; - /* - * CREATE STATISTICS will influence future execution plans - * but does not interfere with currently executing plans. - * So it should be enough to take ShareUpdateExclusiveLock - * on relation, conflicting with ANALYZE and other DDL - * that sets statistical information, but not with normal - * queries. - * - * XXX RangeVarCallbackOwnsRelation not needed here, to - * keep the same behavior as before. - */ - relid = RangeVarGetRelid(rel, ShareUpdateExclusiveLock, false); + /* + * CREATE STATISTICS will influence future execution + * plans but does not interfere with currently + * executing plans. So it should be enough to take + * ShareUpdateExclusiveLock on relation, conflicting + * with ANALYZE and other DDL that sets statistical + * information, but not with normal queries. + * + * XXX RangeVarCallbackOwnsRelation not needed here, + * to keep the same behavior as before. + */ + relid = RangeVarGetRelid(rel, ShareUpdateExclusiveLock, false); + relids = list_make1_oid(relid); - /* Run parse analysis ... */ - stmt = transformStatsStmt(relid, stmt, queryString); + /* Run parse analysis ... */ + stmt = transformStatsStmt(relid, stmt, queryString); + } + else if (IsA(fromNode, JoinExpr)) + { + /* Join statistics passes 0 as relid */ + relid = InvalidOid; + + /* Run parse analysis ... */ + stmt = transformStatsStmt(relid, stmt, queryString); + } + else + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("CREATE STATISTICS only supports relation names or JOIN clauses in the FROM clause"))); - address = CreateStatistics(list_make1_oid(relid), stmt, - true); + address = CreateStatistics(relids, stmt, true); } break; diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c index 58a828ae319..1d4ba386f73 100644 --- a/src/backend/utils/adt/ruleutils.c +++ b/src/backend/utils/adt/ruleutils.c @@ -363,7 +363,8 @@ static char *pg_get_indexdef_worker(Oid indexrelid, int colno, bool showTblSpc, bool inherits, int prettyFlags, bool missing_ok); static char *pg_get_statisticsobj_worker(Oid statextid, - bool missing_ok); + bool missing_ok, + int *on_offset); static char *pg_get_partkeydef_worker(Oid relid, int prettyFlags, bool attrsOnly, bool missing_ok); static char *pg_get_constraintdef_worker(Oid constraintId, bool fullCommand, @@ -1606,20 +1607,35 @@ pg_get_querydef(Query *query, bool pretty) /* * pg_get_statisticsobjdef - * Get the definition of an extended statistics object + * Get the definition of an extended statistics object. + * + * With show_prefix false, the leading "CREATE STATISTICS name (kinds)" is + * omitted and only "ON columns FROM relation(s)" is returned. */ Datum pg_get_statisticsobjdef(PG_FUNCTION_ARGS) { Oid statextid = PG_GETARG_OID(0); + bool show_prefix = PG_GETARG_BOOL(1); char *res; + int on_offset = -1; + text *result; - res = pg_get_statisticsobj_worker(statextid, true); + res = pg_get_statisticsobj_worker(statextid, true, + show_prefix ? NULL : &on_offset); if (res == NULL) PG_RETURN_NULL(); - PG_RETURN_TEXT_P(string_to_text(res)); + /* + * cstring_to_text() copies rather than taking ownership, so this is safe + * even when show_prefix is false and we're handing it a pointer into the + * middle of res; pfree() must still be given the original allocation. + */ + result = cstring_to_text(show_prefix ? res : res + on_offset); + pfree(res); + + PG_RETURN_TEXT_P(result); } /* @@ -1629,7 +1645,7 @@ pg_get_statisticsobjdef(PG_FUNCTION_ARGS) char * pg_get_statisticsobjdef_string(Oid statextid) { - return pg_get_statisticsobj_worker(statextid, false); + return pg_get_statisticsobj_worker(statextid, false, NULL); } /* @@ -1663,8 +1679,11 @@ pg_get_statisticsobjdef_columns(PG_FUNCTION_ARGS) Form_pg_statistic_ext statextrec; HeapTuple statexttup; Datum datum; + bool isnull; List *allexprs = NIL; char *tmp; + oidvector *joinrels = NULL; + int nrels = 0; List *context; ListCell *lc; ArrayBuildState *astate = NULL; @@ -1685,12 +1704,32 @@ pg_get_statisticsobjdef_columns(PG_FUNCTION_ARGS) context = deparse_context_for(get_relation_name(statextrec->stxrelid), statextrec->stxrelid); + /* For join stats, load stxjoinrels to resolve per-column relids */ + datum = SysCacheGetAttr(STATEXTOID, statexttup, + Anum_pg_statistic_ext_stxjoinrels, &isnull); + if (!isnull) + { + /* stxjoinrels already starts with the anchor (stxrelid) */ + joinrels = (oidvector *) DatumGetPointer(datum); + nrels = joinrels->dim1; + } + foreach(lc, allexprs) { Node *expr = (Node *) lfirst(lc); char *str; + Oid col_relid = statextrec->stxrelid; - str = deparse_stat_entry(expr, statextrec->stxrelid, context); + /* For join stats, resolve the correct relid */ + if (joinrels != NULL && IsA(expr, Var)) + { + int varno = ((Var *) expr)->varno; + + if (varno >= 1 && varno <= nrels) + col_relid = joinrels->values[varno - 1]; + } + + str = deparse_stat_entry(expr, col_relid, context); astate = accumArrayResult(astate, PointerGetDatum(cstring_to_text(str)), false, @@ -1710,7 +1749,7 @@ pg_get_statisticsobjdef_columns(PG_FUNCTION_ARGS) * Internal workhorse to decompile an extended statistics object. */ static char * -pg_get_statisticsobj_worker(Oid statextid, bool missing_ok) +pg_get_statisticsobj_worker(Oid statextid, bool missing_ok, int *on_offset) { Form_pg_statistic_ext statextrec; HeapTuple statexttup; @@ -1728,8 +1767,15 @@ pg_get_statisticsobj_worker(Oid statextid, bool missing_ok) List *context; ListCell *lc; List *exprs = NIL; + bool is_join_stat; int ncolumns; + /* Join stat fields */ + oidvector *joinrels = NULL; + List *joinconds = NIL; + char **rel_aliases = NULL; + int nrels = 0; + statexttup = SearchSysCache1(STATEXTOID, ObjectIdGetDatum(statextid)); if (!HeapTupleIsValid(statexttup)) @@ -1741,6 +1787,67 @@ pg_get_statisticsobj_worker(Oid statextid, bool missing_ok) statextrec = (Form_pg_statistic_ext) GETSTRUCT(statexttup); + /* is this a join statistics object? */ + is_join_stat = !heap_attisnull(statexttup, Anum_pg_statistic_ext_stxjoinrels, NULL); + + /* + * For join statistics, load the join-related catalog fields so we can + * emit table-qualified column names and the JOIN ... ON FROM clause. + */ + if (is_join_stat) + { + Datum jdatum; + bool isnull; + char *str; + + jdatum = SysCacheGetAttr(STATEXTOID, statexttup, + Anum_pg_statistic_ext_stxjoinrels, &isnull); + Assert(!isnull); + joinrels = (oidvector *) DatumGetPointer(jdatum); + + jdatum = SysCacheGetAttr(STATEXTOID, statexttup, + Anum_pg_statistic_ext_stxjoinconds, &isnull); + Assert(!isnull); + str = TextDatumGetCString(jdatum); + joinconds = (List *) stringToNode(str); + pfree(str); + + /* + * Build relation OID and alias arrays. stxjoinrels already starts + * with the anchor (== stxrelid), so it lists every participating rel. + */ + nrels = joinrels->dim1; + rel_aliases = palloc(nrels * sizeof(char *)); + for (i = 0; i < joinrels->dim1; i++) + rel_aliases[i] = get_relation_name(joinrels->values[i]); + + /* + * Disambiguate aliases when different relations share the same + * unqualified name (e.g., s1.t and s2.t both get alias "t"). Append + * _1, _2, ... following the set_rtable_names() convention. + */ + for (i = 1; i < nrels; i++) + { + int j; + int suffix = 0; + + for (j = 0; j < i; j++) + { + if (strcmp(rel_aliases[i], rel_aliases[j]) == 0) + { + suffix++; + + rel_aliases[i] = psprintf("%s_%d", + get_relation_name(joinrels->values[i]), + suffix); + + /* Restart scan to check for further conflicts */ + j = -1; + } + } + } + } + /* * Get the statistics expressions. (NOTE: we do not use the relcache * versions of the expressions, because we want to display @@ -1801,7 +1908,7 @@ pg_get_statisticsobj_worker(Oid statextid, bool missing_ok) * an expression statistics. In that case we don't need to specify kinds. */ if ((!ndistinct_enabled || !dependencies_enabled || !mcv_enabled) && - (ncolumns > 1)) + (ncolumns > 1 || is_join_stat)) { bool gotone = false; @@ -1827,6 +1934,9 @@ pg_get_statisticsobj_worker(Oid statextid, bool missing_ok) appendStringInfoString(&buf, " ON "); + if (on_offset != NULL) + *on_offset = buf.len; + context = deparse_context_for(get_relation_name(statextrec->stxrelid), statextrec->stxrelid); @@ -1838,14 +1948,98 @@ pg_get_statisticsobj_worker(Oid statextid, bool missing_ok) if (colno > 0) appendStringInfoString(&buf, ", "); - appendStringInfoString(&buf, - deparse_stat_entry(expr, statextrec->stxrelid, - context)); + if (is_join_stat && IsA(expr, Var) && ((Var *) expr)->varattno > 0) + { + /* Join stat: table-qualify column with its relation's alias */ + Var *var = (Var *) expr; + int relidx = var->varno - 1; + char *attname; + + if (relidx < 0 || relidx >= nrels) + elog(ERROR, "invalid varno %d in join statistics expression", + var->varno); + attname = get_attname(joinrels->values[relidx], + var->varattno, false); + appendStringInfo(&buf, "%s.%s", + quote_identifier(rel_aliases[relidx]), + quote_identifier(attname)); + } + else + { + appendStringInfoString(&buf, + deparse_stat_entry(expr, statextrec->stxrelid, + context)); + } colno++; } - appendStringInfo(&buf, " FROM %s", - generate_relation_name(statextrec->stxrelid, NIL)); + if (is_join_stat) + { + /* + * Emit FROM ... JOIN ... ON syntax for join statistics. + */ + appendStringInfo(&buf, " FROM %s %s", + generate_relation_name(joinrels->values[0], NIL), + quote_identifier(rel_aliases[0])); + + /* + * joinrels->values[0] is the anchor (emitted above); JOIN the rest. + * CREATE STATISTICS stores the tables so that each one after the + * anchor is joined to an earlier one, so attaching each condition to + * the join of its later table gives every join at least one + * condition. + */ + for (i = 1; i < nrels; i++) + { + bool first = true; + + appendStringInfo(&buf, " JOIN %s %s ON (", + generate_relation_name(joinrels->values[i], NIL), + quote_identifier(rel_aliases[i])); + + foreach(lc, joinconds) + { + OpExpr *op = (OpExpr *) lfirst(lc); + Var *lvar; + Var *rvar; + int lrelidx; + int rrelidx; + char *lcolname; + char *rcolname; + + if (!join_stats_cond_vars((Node *) op, &lvar, &rvar)) + elog(ERROR, "unexpected join statistics condition"); + lrelidx = lvar->varno - 1; + rrelidx = rvar->varno - 1; + if (Max(lvar->varno, rvar->varno) != i + 1) + continue; + if (lrelidx < 0 || lrelidx >= nrels || + rrelidx < 0 || rrelidx >= nrels) + elog(ERROR, "invalid varno in join statistics condition"); + lcolname = get_attname(joinrels->values[lrelidx], + lvar->varattno, false); + rcolname = get_attname(joinrels->values[rrelidx], + rvar->varattno, false); + appendStringInfo(&buf, "%s%s.%s %s %s.%s", + first ? "" : " AND ", + quote_identifier(rel_aliases[lrelidx]), + quote_identifier(lcolname), + generate_operator_name(op->opno, + exprType(linitial(op->args)), + exprType(lsecond(op->args))), + quote_identifier(rel_aliases[rrelidx]), + quote_identifier(rcolname)); + first = false; + } + + appendStringInfoChar(&buf, ')'); + } + } + else + { + appendStringInfo(&buf, " FROM %s", + generate_relation_name(statextrec->stxrelid, NIL)); + } ReleaseSysCache(statexttup); diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 5f9f00974df..ae99a8d1f6e 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -2686,8 +2686,14 @@ describeOneTableDetails(const char *schemaname, " " CppAsString2(STATS_EXT_NDISTINCT) " = any(stxkind) AS ndist_enabled,\n" " " CppAsString2(STATS_EXT_DEPENDENCIES) " = any(stxkind) AS deps_enabled,\n" " " CppAsString2(STATS_EXT_MCV) " = any(stxkind) AS mcv_enabled,\n" - "stxstattarget\n" - "FROM pg_catalog.pg_statistic_ext\n" + "stxstattarget,\n"); + if (pset.sversion >= 200000) + appendPQExpBufferStr(&buf, + "pg_catalog.pg_get_statisticsobjdef(oid, false) AS body\n"); + else + appendPQExpBufferStr(&buf, + "NULL AS body\n"); + appendPQExpBuffer(&buf, "FROM pg_catalog.pg_statistic_ext\n" "WHERE stxrelid = '%s'\n" "ORDER BY nsp, stxname;", oid); @@ -2758,9 +2764,18 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferChar(&buf, ')'); } - appendPQExpBuffer(&buf, " ON %s FROM %s", - PQgetvalue(result, i, 4), - PQgetvalue(result, i, 1)); + /* + * Use pg_get_statisticsobjdef(oid, false) when available + * (PG20+); fall back to simple column list for older + * servers. + */ + if (!PQgetisnull(result, i, 9)) + appendPQExpBuffer(&buf, " ON %s", + PQgetvalue(result, i, 9)); + else + appendPQExpBuffer(&buf, " ON %s FROM %s", + PQgetvalue(result, i, 4), + PQgetvalue(result, i, 1)); /* Show the stats target if it's not default */ if (!PQgetisnull(result, i, 8) && @@ -2790,9 +2805,15 @@ describeOneTableDetails(const char *schemaname, " " CppAsString2(STATS_EXT_MCV) " = any(stxkind) AS mcv_enabled,\n"); if (pset.sversion >= 130000) - appendPQExpBufferStr(&buf, " stxstattarget\n"); + appendPQExpBufferStr(&buf, " stxstattarget,\n"); + else + appendPQExpBufferStr(&buf, " -1 AS stxstattarget,\n"); + if (pset.sversion >= 200000) + appendPQExpBufferStr(&buf, + " pg_catalog.pg_get_statisticsobjdef(oid, false) AS body\n"); else - appendPQExpBufferStr(&buf, " -1 AS stxstattarget\n"); + appendPQExpBufferStr(&buf, + " NULL AS body\n"); appendPQExpBuffer(&buf, "FROM pg_catalog.pg_statistic_ext\n" "WHERE stxrelid = '%s'\n" "ORDER BY 1;", @@ -2837,9 +2858,21 @@ describeOneTableDetails(const char *schemaname, appendPQExpBuffer(&buf, "%smcv", gotone ? ", " : ""); } - appendPQExpBuffer(&buf, ") ON %s FROM %s", - PQgetvalue(result, i, 4), - PQgetvalue(result, i, 1)); + /* + * Use pg_get_statisticsobjdef(oid, false) when available + * (PG20+); fall back to simple column list for older + * servers. + */ + if (!PQgetisnull(result, i, 9)) + { + appendPQExpBufferChar(&buf, ')'); + appendPQExpBuffer(&buf, " ON %s", + PQgetvalue(result, i, 9)); + } + else + appendPQExpBuffer(&buf, ") ON %s FROM %s", + PQgetvalue(result, i, 4), + PQgetvalue(result, i, 1)); /* Show the stats target if it's not default */ if (strcmp(PQgetvalue(result, i, 8), "-1") != 0) @@ -4839,9 +4872,7 @@ listExtendedStats(const char *pattern, bool verbose) if (pset.sversion >= 200000) appendPQExpBuffer(&buf, - "pg_catalog.format('%%s FROM %%s', \n" - " pg_catalog.array_to_string(pg_catalog.pg_get_statisticsobjdef_columns(es.oid), ', '), \n" - " es.stxrelid::pg_catalog.regclass) AS \"%s\"", + "pg_catalog.pg_get_statisticsobjdef(es.oid, false) AS \"%s\"", gettext_noop("Definition")); else if (pset.sversion >= 140000) appendPQExpBuffer(&buf, diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 741e1d45f90..24ece4eb3eb 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -3993,7 +3993,8 @@ proargtypes => 'oid', prosrc => 'pg_get_indexdef' }, { oid => '3415', descr => 'extended statistics object description', proname => 'pg_get_statisticsobjdef', provolatile => 's', - prorettype => 'text', proargtypes => 'oid', + prorettype => 'text', proargtypes => 'oid bool', + proargnames => '{statextid,show_prefix}', proargdefaults => '{true}', prosrc => 'pg_get_statisticsobjdef' }, { oid => '6174', descr => 'extended statistics columns and expressions', proname => 'pg_get_statisticsobjdef_columns', provolatile => 's', diff --git a/src/include/catalog/pg_statistic_ext.h b/src/include/catalog/pg_statistic_ext.h index 08ed6e23287..fbff51214fd 100644 --- a/src/include/catalog/pg_statistic_ext.h +++ b/src/include/catalog/pg_statistic_ext.h @@ -55,6 +55,12 @@ CATALOG(pg_statistic_ext,3381,StatisticExtRelationId) * statistics object is * defined on; simple columns * are Var nodes */ + + /* Fields for join statistics (all NULL for single-table stats) */ + + /* participating rels anchor-first (stxjoinrels[0] == stxrelid) */ + oidvector stxjoinrels BKI_DEFAULT(_null_) BKI_FORCE_NULL; + pg_node_tree stxjoinconds BKI_DEFAULT(_null_) BKI_FORCE_NULL; #endif } FormData_pg_statistic_ext; @@ -74,6 +80,9 @@ DECLARE_UNIQUE_INDEX_PKEY(pg_statistic_ext_oid_index, 3380, StatisticExtOidIndex DECLARE_UNIQUE_INDEX(pg_statistic_ext_name_index, 3997, StatisticExtNameIndexId, pg_statistic_ext, btree(stxname name_ops, stxnamespace oid_ops)); DECLARE_INDEX(pg_statistic_ext_relid_index, 3379, StatisticExtRelidIndexId, pg_statistic_ext, btree(stxrelid oid_ops)); +/* each participating relation of a join statistics object exists */ +DECLARE_ARRAY_FOREIGN_KEY((stxjoinrels), pg_class, (oid)); + MAKE_SYSCACHE(STATEXTOID, pg_statistic_ext_oid_index, 4); MAKE_SYSCACHE(STATEXTNAMENSP, pg_statistic_ext_name_index, 4); diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 5be1d4a944b..fb43eff65b1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -3574,6 +3574,12 @@ typedef struct CreateStatsStmt bool transformed; /* true when transformStatsStmt is finished */ bool if_not_exists; /* do nothing if stats name already exists */ Oid owner; /* OID of owner, or InvalidOid for default */ + + /* Join statistics fields, populated by transformStatsStmt */ + Oid stxrelid; /* anchor table OID (InvalidOid if not join) */ + List *stxjoinrels; /* participating table OIDs, anchor first + * (list of Oid) */ + List *stxjoinconds; /* equijoin conditions (list of OpExpr) */ } CreateStatsStmt; /* diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h index 2f63d7b2b59..3aca60baff6 100644 --- a/src/include/nodes/pathnodes.h +++ b/src/include/nodes/pathnodes.h @@ -642,6 +642,8 @@ struct PlannerInfo bool placeholdersFrozen; /* true if planning a recursive WITH item */ bool hasRecursion; + /* true if any base relation has join statistics */ + bool hasJoinStats; /* true if a planner extension may replan this subquery */ bool assumeReplanning; @@ -1533,6 +1535,16 @@ typedef struct StatisticExtInfo /* positions in "exprs" of the entries that are plain columns */ Bitmapset *columns; + + /* + * Join statistics fields (all NIL for single-table stats). For a join + * stat the columns in "exprs" above carry stxjoinrels-relative varnos (1 + * = anchor, 2 = other joined relation), which the join estimator uses to + * map each column to its relation. + */ + List *joinrels; /* OIDs of participating relations, anchor + * first */ + List *joinconds; /* parsed join conditions (List of OpExpr) */ } StatisticExtInfo; /* diff --git a/src/include/optimizer/joininfo.h b/src/include/optimizer/joininfo.h index aa26cc670f9..6debaabc1cc 100644 --- a/src/include/optimizer/joininfo.h +++ b/src/include/optimizer/joininfo.h @@ -19,6 +19,8 @@ extern bool have_relevant_joinclause(PlannerInfo *root, RelOptInfo *rel1, RelOptInfo *rel2); +extern int count_relevant_joinclauses(PlannerInfo *root, + RelOptInfo *rel1, RelOptInfo *rel2); extern void add_join_clause_to_rels(PlannerInfo *root, RestrictInfo *restrictinfo, diff --git a/src/include/optimizer/paths.h b/src/include/optimizer/paths.h index 64f87da1674..14cb9d54b94 100644 --- a/src/include/optimizer/paths.h +++ b/src/include/optimizer/paths.h @@ -202,6 +202,9 @@ extern List *generate_implied_equalities_for_column(PlannerInfo *root, Relids prohibited_rels); extern bool have_relevant_eclass_joinclause(PlannerInfo *root, RelOptInfo *rel1, RelOptInfo *rel2); +extern int count_relevant_eclass_joinclauses(PlannerInfo *root, + RelOptInfo *rel1, + RelOptInfo *rel2); extern bool has_relevant_eclass_joinclause(PlannerInfo *root, RelOptInfo *rel1); extern bool eclass_useful_for_merging(PlannerInfo *root, diff --git a/src/include/statistics/extended_stats_internal.h b/src/include/statistics/extended_stats_internal.h index 844b149a7cb..08df7656b6e 100644 --- a/src/include/statistics/extended_stats_internal.h +++ b/src/include/statistics/extended_stats_internal.h @@ -68,7 +68,6 @@ typedef struct StatsBuildData bool **nulls; } StatsBuildData; - extern MVNDistinct *statext_ndistinct_build(double totalrows, StatsBuildData *data); extern bytea *statext_ndistinct_serialize(MVNDistinct *ndistinct); extern MVNDistinct *statext_ndistinct_deserialize(bytea *data); @@ -96,6 +95,22 @@ extern Datum statext_mcv_import(int elevel, int numattrs, Oid *atttypids, bool *mcv_nulls, float8 *freqs, float8 *base_freqs); +extern Oid find_index_for_operator(Relation rel, AttrNumber attnum, + Oid eq_op); + +extern MCVList *statext_join_mcv_build(Relation anchor_rel, + Oid *joinrel_oids, + int njoinrels, + List *joinconds, + Oid index_oid, + int16 *stxkeys, + int16 *keyrefs, + int nkeys, + int stattarget, + int numrows, HeapTuple *rows, + double totalrows, + VacAttrStats ***result_stats); + extern MultiSortSupport multi_sort_init(int ndims); extern void multi_sort_add_dimension(MultiSortSupport mss, int sortdim, Oid oper, Oid collation); @@ -112,9 +127,18 @@ extern SortItem *build_sorted_items(StatsBuildData *data, int *nitems, MultiSortSupport mss, int numattrs, AttrNumber *attnums); +extern bool statext_is_compatible_clause_internal(PlannerInfo *root, + Node *clause, Index relid, + Bitmapset **attnums, + List **exprs, + bool *leakproof); + extern bool examine_opclause_args(List *args, Node **exprp, Const **cstp, bool *expronleftp); +extern bool *mcv_get_match_bitmap(PlannerInfo *root, List *clauses, + List *exprs, MCVList *mcvlist, bool is_or); + extern Selectivity mcv_combine_selectivities(Selectivity simple_sel, Selectivity mcv_sel, Selectivity mcv_basesel, diff --git a/src/include/statistics/statistics.h b/src/include/statistics/statistics.h index 888d3e27cc7..04451a81989 100644 --- a/src/include/statistics/statistics.h +++ b/src/include/statistics/statistics.h @@ -159,4 +159,18 @@ extern bool import_attribute_statistics(Relation rel, extern bool delete_attribute_statistics(Relation rel, AttrNumber attnum, bool inherited); +/* Join MCV statistics functions */ +extern bool join_stats_cond_vars(Node *cond, Var **lvar, Var **rvar); +extern void join_stats_cond_sides(OpExpr *cond, Var **sampled, Var **probed, + bool *commuted); +extern Selectivity join_mcv_join_selectivity(PlannerInfo *root, + Relids outer_relids, + Relids inner_relids, + List *clauses, + Selectivity join_sel); +extern Selectivity join_mcv_restriction_selectivity(PlannerInfo *root, + RestrictInfo *rinfo, + int varRelid, + Selectivity sel); + #endif /* STATISTICS_H */ diff --git a/src/test/isolation/expected/join-stats-analyze.out b/src/test/isolation/expected/join-stats-analyze.out new file mode 100644 index 00000000000..7d98ad91f16 --- /dev/null +++ b/src/test/isolation/expected/join-stats-analyze.out @@ -0,0 +1,12 @@ +Parsed test spec with 2 sessions + +starting permutation: lock_dim analyze_fact alter_dim commit +step lock_dim: + BEGIN; + LOCK dim IN ACCESS EXCLUSIVE MODE; + +s2: WARNING: skipping join statistics "public.join_stat" +DETAIL: Could not obtain lock on table "public.dim". +step analyze_fact: ANALYZE fact; +step alter_dim: ALTER TABLE dim ALTER COLUMN label TYPE varchar(20); +step commit: COMMIT; diff --git a/src/test/isolation/isolation_schedule b/src/test/isolation/isolation_schedule index f448e4d6e13..c03955c6dd9 100644 --- a/src/test/isolation/isolation_schedule +++ b/src/test/isolation/isolation_schedule @@ -111,6 +111,7 @@ test: timeouts test: vacuum-concurrent-drop test: vacuum-conflict test: vacuum-skip-locked +test: join-stats-analyze test: stats test: horizons test: predicate-hash diff --git a/src/test/isolation/specs/join-stats-analyze.spec b/src/test/isolation/specs/join-stats-analyze.spec new file mode 100644 index 00000000000..ede62b7ad14 --- /dev/null +++ b/src/test/isolation/specs/join-stats-analyze.spec @@ -0,0 +1,41 @@ +# ANALYZE of join statistics while the joined table is locked. +# +# ANALYZE of fact holds ShareUpdateExclusiveLock on fact and then needs +# AccessShareLock on dim to sample the join. ALTER TABLE dim ALTER COLUMN +# TYPE holds AccessExclusiveLock on dim and then needs ShareUpdateExclusiveLock +# on fact to re-create the statistics. ANALYZE must not wait for dim, or the +# two deadlock. + +setup +{ + CREATE TABLE dim (id int PRIMARY KEY, label text); + CREATE TABLE fact (dim_id int) WITH (autovacuum_enabled = false); + ALTER TABLE dim SET (autovacuum_enabled = false); + INSERT INTO dim SELECT g, 'label ' || g % 3 FROM generate_series(1, 100) g; + INSERT INTO fact SELECT 1 + g % 100 FROM generate_series(1, 1000) g; + CREATE STATISTICS join_stat (mcv) ON dim.label + FROM fact JOIN dim ON (fact.dim_id = dim.id); +} + +teardown +{ + DROP TABLE fact, dim; +} + +# If ANALYZE waited for dim, this session would detect the deadlock, as in +# the reported case +session s1 +setup { SET deadlock_timeout = '10ms'; } +step lock_dim +{ + BEGIN; + LOCK dim IN ACCESS EXCLUSIVE MODE; +} +step alter_dim { ALTER TABLE dim ALTER COLUMN label TYPE varchar(20); } +step commit { COMMIT; } + +session s2 +setup { SET deadlock_timeout = '10s'; } +step analyze_fact { ANALYZE fact; } + +permutation lock_dim analyze_fact alter_dim commit diff --git a/src/test/regress/expected/oidjoins.out b/src/test/regress/expected/oidjoins.out index 419da49663a..af70b9dd991 100644 --- a/src/test/regress/expected/oidjoins.out +++ b/src/test/regress/expected/oidjoins.out @@ -166,6 +166,7 @@ NOTICE: checking pg_statistic {starelid,staattnum} => pg_attribute {attrelid,at NOTICE: checking pg_statistic_ext {stxrelid} => pg_class {oid} NOTICE: checking pg_statistic_ext {stxnamespace} => pg_namespace {oid} NOTICE: checking pg_statistic_ext {stxowner} => pg_authid {oid} +NOTICE: checking pg_statistic_ext {stxjoinrels} => pg_class {oid} NOTICE: checking pg_statistic_ext_data {stxoid} => pg_statistic_ext {oid} NOTICE: checking pg_rewrite {ev_class} => pg_class {oid} NOTICE: checking pg_trigger {tgrelid} => pg_class {oid} diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index 9a144be96a4..80e7108588b 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -2741,7 +2741,10 @@ pg_stats_ext| SELECT cn.nspname AS schemaname, array_agg(pg_mcv_list_items.frequency) AS most_common_freqs, array_agg(pg_mcv_list_items.base_frequency) AS most_common_base_freqs FROM pg_mcv_list_items(sd.stxdmcv) pg_mcv_list_items(index, "values", nulls, frequency, base_frequency)) m ON ((sd.stxdmcv IS NOT NULL))) - WHERE (pg_has_role(c.relowner, 'USAGE'::text) AND ((c.relrowsecurity = false) OR (NOT row_security_active(c.oid)))); + WHERE (pg_has_role(c.relowner, 'USAGE'::text) AND ((s.stxjoinrels IS NULL) OR (NOT (EXISTS ( SELECT 1 + FROM (unnest(s.stxjoinrels) jr(oid) + JOIN pg_class jc ON ((jc.oid = jr.oid))) + WHERE ((NOT pg_has_role(jc.relowner, 'USAGE'::text)) OR (jc.relrowsecurity AND row_security_active(jc.oid))))))) AND ((c.relrowsecurity = false) OR (NOT row_security_active(c.oid)))); pg_stats_ext_exprs| SELECT cn.nspname AS schemaname, c.relname AS tablename, s.stxrelid AS tableid, @@ -2841,7 +2844,10 @@ pg_stats_ext_exprs| SELECT cn.nspname AS schemaname, LEFT JOIN pg_namespace sn ON ((sn.oid = s.stxnamespace))) JOIN LATERAL ( SELECT unnest(pg_get_statisticsobjdef_expressions(s.oid)) AS expr, unnest(sd.stxdexpr) AS a) stat ON ((stat.expr IS NOT NULL))) - WHERE (pg_has_role(c.relowner, 'USAGE'::text) AND ((c.relrowsecurity = false) OR (NOT row_security_active(c.oid)))); + WHERE (pg_has_role(c.relowner, 'USAGE'::text) AND ((s.stxjoinrels IS NULL) OR (NOT (EXISTS ( SELECT 1 + FROM (unnest(s.stxjoinrels) jr(oid) + JOIN pg_class jc ON ((jc.oid = jr.oid))) + WHERE ((NOT pg_has_role(jc.relowner, 'USAGE'::text)) OR (jc.relrowsecurity AND row_security_active(jc.oid))))))) AND ((c.relrowsecurity = false) OR (NOT row_security_active(c.oid)))); pg_tables| SELECT n.nspname AS schemaname, c.relname AS tablename, pg_get_userbyid(c.relowner) AS tableowner, diff --git a/src/test/regress/expected/stats_ext.out b/src/test/regress/expected/stats_ext.out index 0af4144b60e..0b7cae90417 100644 --- a/src/test/regress/expected/stats_ext.out +++ b/src/test/regress/expected/stats_ext.out @@ -56,26 +56,26 @@ CREATE STATISTICS tst (unrecognized) ON x, y FROM ext_stats_test; ERROR: unrecognized statistics kind "unrecognized" -- unsupported targets CREATE STATISTICS tst ON x, y FROM ext_stats_test, ext_stats_test; -ERROR: only a single relation is allowed in CREATE STATISTICS +ERROR: only a single relation or JOIN is allowed in CREATE STATISTICS CREATE STATISTICS tst ON x, y FROM ext_stats_test, (SELECT * FROM ext_stats_test) AS foo; -ERROR: only a single relation is allowed in CREATE STATISTICS +ERROR: only a single relation or JOIN is allowed in CREATE STATISTICS CREATE STATISTICS tst ON a FROM (VALUES (x)) AS foo; -ERROR: CREATE STATISTICS only supports relation names in the FROM clause +ERROR: CREATE STATISTICS only supports relation names or JOIN clauses in the FROM clause CREATE STATISTICS tst ON a FROM foo NATURAL JOIN bar; -ERROR: CREATE STATISTICS only supports relation names in the FROM clause +ERROR: relation "foo" does not exist CREATE STATISTICS tst ON a FROM (SELECT * FROM ext_stats_test) AS foo; -ERROR: CREATE STATISTICS only supports relation names in the FROM clause +ERROR: CREATE STATISTICS only supports relation names or JOIN clauses in the FROM clause CREATE STATISTICS tst ON a FROM ext_stats_test s TABLESAMPLE system (x); -ERROR: CREATE STATISTICS only supports relation names in the FROM clause +ERROR: CREATE STATISTICS only supports relation names or JOIN clauses in the FROM clause CREATE STATISTICS tst ON a FROM XMLTABLE('foo' PASSING 'bar' COLUMNS a text); -ERROR: CREATE STATISTICS only supports relation names in the FROM clause +ERROR: CREATE STATISTICS only supports relation names or JOIN clauses in the FROM clause CREATE STATISTICS tst ON a FROM JSON_TABLE(jsonb '123', '$' COLUMNS (item int)); -ERROR: CREATE STATISTICS only supports relation names in the FROM clause +ERROR: CREATE STATISTICS only supports relation names or JOIN clauses in the FROM clause CREATE FUNCTION tftest(int) returns table(a int, b int) as $$ SELECT $1, $1+i FROM generate_series(1,5) g(i); $$ LANGUAGE sql IMMUTABLE STRICT; CREATE STATISTICS alt_stat2 ON a FROM tftest(1); -ERROR: CREATE STATISTICS only supports relation names in the FROM clause +ERROR: CREATE STATISTICS only supports relation names or JOIN clauses in the FROM clause DROP FUNCTION tftest; -- incorrect expressions CREATE STATISTICS tst ON (y) FROM ext_stats_test; -- single column reference @@ -3967,11 +3967,13 @@ DROP TABLE declared_order_grouping; -- Auto-generated statistics object names. -- -- A simple column reference contributes the column name, whether written --- unqualified or wrapped in parens; a more complex expression contributes a --- fixed "expr". +-- unqualified, table-qualified, or wrapped in parens; a more complex expression +-- contributes a fixed "expr". CREATE TABLE auto_t1 (a INTEGER, b INTEGER, c INTEGER); -- Unqualified columns: bare column names. CREATE STATISTICS ON b, c FROM auto_t1; +-- Table-qualified columns: still the column names. +CREATE STATISTICS ON auto_t1.a, auto_t1.b FROM auto_t1; -- Table-qualified columns in parens: still the column names. CREATE STATISTICS ON (auto_t1.a), (auto_t1.b) FROM auto_t1; -- A parenthesized column and a more complex expression: the column resolves @@ -3982,8 +3984,9 @@ CREATE STATISTICS ON (a), (b + c) FROM auto_t1; Schema | Name | Definition | Ndistinct | Dependencies | MCV --------+---------------------+-------------------------+-----------+--------------+--------- public | auto_t1_a_b_stat | a, b FROM auto_t1 | defined | defined | defined + public | auto_t1_a_b_stat1 | a, b FROM auto_t1 | defined | defined | defined public | auto_t1_a_expr_stat | a, (b + c) FROM auto_t1 | defined | defined | defined public | auto_t1_b_c_stat | b, c FROM auto_t1 | defined | defined | defined -(3 rows) +(4 rows) DROP TABLE auto_t1; diff --git a/src/test/regress/expected/stats_ext_crossrel.out b/src/test/regress/expected/stats_ext_crossrel.out new file mode 100644 index 00000000000..9751c90fa58 --- /dev/null +++ b/src/test/regress/expected/stats_ext_crossrel.out @@ -0,0 +1,3058 @@ +-- Join MCV statistics +-- +-- Note: tables for which we check estimated row counts should be created +-- with autovacuum_enabled = off, so that we don't have unstable results +-- from auto-analyze happening when we didn't expect it. +-- +-- Each movie has one keyword out of 50: a few keywords are on many movies +-- and most on few or none, which the per-column statistics do not see. +-- Each keyword has its own phonetic_code, NULL for some, and the keyword +-- decides the year. +CREATE TABLE join_mcv_keywords ( + id INTEGER PRIMARY KEY, + keyword TEXT NOT NULL, + phonetic_code character varying(5) +) WITH (autovacuum_enabled = off); +-- keyword_id has no foreign key to join_mcv_keywords: the join is estimated +-- from the columns' statistics +CREATE TABLE join_mcv_movies ( + movie_id INTEGER PRIMARY KEY, + keyword_id INTEGER NOT NULL, + year INTEGER NOT NULL +) WITH (autovacuum_enabled = off); +INSERT INTO join_mcv_keywords (id, keyword, phonetic_code) +SELECT + i, + 'keyword_' || i, + CASE WHEN i BETWEEN 26 AND 30 THEN NULL ELSE 'ph_' || i END +FROM generate_series(1, 50) i; +INSERT INTO join_mcv_movies (movie_id, keyword_id, year) +SELECT + i, + CASE + WHEN i % 100 < 60 THEN (i % 10) + 1 + WHEN i % 100 < 90 THEN (i % 10) + 11 + ELSE (i % 10) + 21 + END, + CASE + WHEN i % 100 < 60 THEN 2020 + WHEN i % 100 < 90 THEN 2021 + ELSE 2022 + END +FROM generate_series(1, 10000) i; +ANALYZE join_mcv_keywords, join_mcv_movies; +-- The first table of the FROM clause is the anchor: the statistics object +-- belongs to it, and ANALYZE on it samples the join. The definition as +-- stored: the anchor in stxrelid, every table in stxjoinrels, and the mcv +-- kind +CREATE STATISTICS join_mcv_movies_keyword_stats (mcv) +ON k.keyword +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword_id = k.id); +SELECT stxrelid::regclass, stxjoinrels::regclass[], stxkind +FROM pg_statistic_ext WHERE stxname = 'join_mcv_movies_keyword_stats'; + stxrelid | stxjoinrels | stxkind +-----------------+-------------------------------------------+--------- + join_mcv_movies | [0:1]={join_mcv_movies,join_mcv_keywords} | {m} +(1 row) + +SELECT pg_get_statisticsobjdef(oid, false) FROM pg_statistic_ext +WHERE stxname = 'join_mcv_movies_keyword_stats'; + pg_get_statisticsobjdef +---------------------------------------------------------------------------------------------------------------------------------------------------------------- + join_mcv_keywords.keyword FROM join_mcv_movies join_mcv_movies JOIN join_mcv_keywords join_mcv_keywords ON (join_mcv_movies.keyword_id = join_mcv_keywords.id) +(1 row) + +\dX+ join_mcv_movies_keyword_stats + List of extended statistics + Schema | Name | Definition | Ndistinct | Dependencies | MCV | Description +--------+-------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+--------------+---------+------------- + public | join_mcv_movies_keyword_stats | join_mcv_keywords.keyword FROM join_mcv_movies join_mcv_movies JOIN join_mcv_keywords join_mcv_keywords ON (join_mcv_movies.keyword_id = join_mcv_keywords.id) | | | defined | +(1 row) + +-- the anchor table lists its join statistics +\d+ join_mcv_movies + Table "public.join_mcv_movies" + Column | Type | Collation | Nullable | Default | Storage | Stats target | Description +------------+---------+-----------+----------+---------+---------+--------------+------------- + movie_id | integer | | not null | | plain | | + keyword_id | integer | | not null | | plain | | + year | integer | | not null | | plain | | +Indexes: + "join_mcv_movies_pkey" PRIMARY KEY, btree (movie_id) +Statistics objects: + "public.join_mcv_movies_keyword_stats" (mcv) ON join_mcv_keywords.keyword FROM join_mcv_movies join_mcv_movies JOIN join_mcv_keywords join_mcv_keywords ON (join_mcv_movies.keyword_id = join_mcv_keywords.id) +Not-null constraints: + "join_mcv_movies_movie_id_not_null" NOT NULL "movie_id" + "join_mcv_movies_keyword_id_not_null" NOT NULL "keyword_id" + "join_mcv_movies_year_not_null" NOT NULL "year" +Options: autovacuum_enabled=off + +-- pg_stats_ext resolves the column in join_mcv_keywords, not by attribute +-- number in the anchor, where 2 is keyword_id +ANALYZE join_mcv_movies; +SELECT exprs FROM pg_stats_ext +WHERE statistics_name = 'join_mcv_movies_keyword_stats'; + exprs +----------- + {keyword} +(1 row) + +DROP STATISTICS join_mcv_movies_keyword_stats; +-- without join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_movies mk, join_mcv_keywords k + WHERE k.keyword = ''keyword_1'' AND k.id = mk.keyword_id +'); + estimated | actual +-----------+-------- + 200 | 600 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_movies mk, join_mcv_keywords k + WHERE k.keyword IN (''keyword_1'', ''keyword_2'', ''keyword_3'') + AND k.id = mk.keyword_id +'); + estimated | actual +-----------+-------- + 600 | 1800 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_movies mk, join_mcv_keywords k + WHERE k.keyword = ''keyword_1'' + AND k.phonetic_code = ''ph_15'' + AND k.id = mk.keyword_id +'); + estimated | actual +-----------+-------- + 200 | 0 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_movies mk, join_mcv_keywords k + WHERE k.phonetic_code = ''ph_1'' AND k.id = mk.keyword_id +'); + estimated | actual +-----------+-------- + 200 | 600 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_movies mk, join_mcv_keywords k + WHERE k.keyword = ''keyword_1'' AND mk.year = 2020 + AND k.id = mk.keyword_id +'); + estimated | actual +-----------+-------- + 120 | 600 +(1 row) + +-- join statistics on the keyword: each joined keyword with its fraction of +-- the 10000 joined rows +CREATE STATISTICS join_mcv_movies_keyword_stats (mcv) +ON k.keyword +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword_id = k.id); +ANALYZE join_mcv_movies; +SELECT m.values, + m.nulls, + ROUND(m.frequency::numeric, 2) AS frequency +FROM pg_statistic_ext s +JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +CROSS JOIN LATERAL pg_mcv_list_items(d.stxdmcv) AS m +WHERE s.stxname = 'join_mcv_movies_keyword_stats' +ORDER BY m.frequency DESC, m.values; + values | nulls | frequency +--------------+-------+----------- + {keyword_1} | {f} | 0.06 + {keyword_10} | {f} | 0.06 + {keyword_2} | {f} | 0.06 + {keyword_3} | {f} | 0.06 + {keyword_4} | {f} | 0.06 + {keyword_5} | {f} | 0.06 + {keyword_6} | {f} | 0.06 + {keyword_7} | {f} | 0.06 + {keyword_8} | {f} | 0.06 + {keyword_9} | {f} | 0.06 + {keyword_11} | {f} | 0.03 + {keyword_12} | {f} | 0.03 + {keyword_13} | {f} | 0.03 + {keyword_14} | {f} | 0.03 + {keyword_15} | {f} | 0.03 + {keyword_16} | {f} | 0.03 + {keyword_17} | {f} | 0.03 + {keyword_18} | {f} | 0.03 + {keyword_19} | {f} | 0.03 + {keyword_20} | {f} | 0.03 + {keyword_21} | {f} | 0.01 + {keyword_22} | {f} | 0.01 + {keyword_23} | {f} | 0.01 + {keyword_24} | {f} | 0.01 + {keyword_25} | {f} | 0.01 + {keyword_26} | {f} | 0.01 + {keyword_27} | {f} | 0.01 + {keyword_28} | {f} | 0.01 + {keyword_29} | {f} | 0.01 + {keyword_30} | {f} | 0.01 +(30 rows) + +-- A stored keyword is estimated from its frequency, alone and in a list +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_movies mk, join_mcv_keywords k + WHERE k.keyword = ''keyword_1'' AND k.id = mk.keyword_id +'); + estimated | actual +-----------+-------- + 600 | 600 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_movies mk, join_mcv_keywords k + WHERE k.keyword IN (''keyword_1'', ''keyword_2'', ''keyword_3'') + AND k.id = mk.keyword_id +'); + estimated | actual +-----------+-------- + 1800 | 1800 +(1 row) + +-- A filter the statistics do not cover is estimated as without them +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_movies mk, join_mcv_keywords k + WHERE k.phonetic_code = ''ph_1'' AND k.id = mk.keyword_id +'); + estimated | actual +-----------+-------- + 200 | 600 +(1 row) + +-- Join statistics on keyword and phonetic_code store every combination in +-- the join +CREATE STATISTICS join_mcv_movies_keyword_code_stats (mcv) +ON k.keyword, k.phonetic_code +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword_id = k.id); +ANALYZE join_mcv_movies; +SELECT m.values, + m.nulls, + ROUND(m.frequency::numeric, 2) AS frequency +FROM pg_statistic_ext s +JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +CROSS JOIN LATERAL pg_mcv_list_items(d.stxdmcv) AS m +WHERE s.stxname = 'join_mcv_movies_keyword_code_stats' +ORDER BY m.frequency DESC, m.values; + values | nulls | frequency +--------------------+-------+----------- + {keyword_1,ph_1} | {f,f} | 0.06 + {keyword_10,ph_10} | {f,f} | 0.06 + {keyword_2,ph_2} | {f,f} | 0.06 + {keyword_3,ph_3} | {f,f} | 0.06 + {keyword_4,ph_4} | {f,f} | 0.06 + {keyword_5,ph_5} | {f,f} | 0.06 + {keyword_6,ph_6} | {f,f} | 0.06 + {keyword_7,ph_7} | {f,f} | 0.06 + {keyword_8,ph_8} | {f,f} | 0.06 + {keyword_9,ph_9} | {f,f} | 0.06 + {keyword_11,ph_11} | {f,f} | 0.03 + {keyword_12,ph_12} | {f,f} | 0.03 + {keyword_13,ph_13} | {f,f} | 0.03 + {keyword_14,ph_14} | {f,f} | 0.03 + {keyword_15,ph_15} | {f,f} | 0.03 + {keyword_16,ph_16} | {f,f} | 0.03 + {keyword_17,ph_17} | {f,f} | 0.03 + {keyword_18,ph_18} | {f,f} | 0.03 + {keyword_19,ph_19} | {f,f} | 0.03 + {keyword_20,ph_20} | {f,f} | 0.03 + {keyword_21,ph_21} | {f,f} | 0.01 + {keyword_22,ph_22} | {f,f} | 0.01 + {keyword_23,ph_23} | {f,f} | 0.01 + {keyword_24,ph_24} | {f,f} | 0.01 + {keyword_25,ph_25} | {f,f} | 0.01 + {keyword_26,NULL} | {f,t} | 0.01 + {keyword_27,NULL} | {f,t} | 0.01 + {keyword_28,NULL} | {f,t} | 0.01 + {keyword_29,NULL} | {f,t} | 0.01 + {keyword_30,NULL} | {f,t} | 0.01 +(30 rows) + +-- phonetic_code is covered now +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_movies mk, join_mcv_keywords k + WHERE k.phonetic_code = ''ph_1'' AND k.id = mk.keyword_id +'); + estimated | actual +-----------+-------- + 600 | 600 +(1 row) + +-- Both columns +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_movies mk, join_mcv_keywords k + WHERE k.keyword = ''keyword_1'' + AND k.phonetic_code = ''ph_1'' + AND k.id = mk.keyword_id +'); + estimated | actual +-----------+-------- + 600 | 600 +(1 row) + +-- keyword_1 never pairs with ph_15, and the statistics store every +-- combination in the join, so the estimate is the one-row minimum +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_movies mk, join_mcv_keywords k + WHERE k.keyword = ''keyword_1'' + AND k.phonetic_code = ''ph_15'' + AND k.id = mk.keyword_id +'); + estimated | actual +-----------+-------- + 1 | 0 +(1 row) + +-- Still no rows with a filter the statistics do not cover added +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_movies mk, join_mcv_keywords k + WHERE k.keyword = ''keyword_1'' + AND k.phonetic_code = ''ph_15'' + AND mk.year = 2020 + AND k.id = mk.keyword_id +'); + estimated | actual +-----------+-------- + 1 | 0 +(1 row) + +-- Join statistics with a column from each table +CREATE STATISTICS join_mcv_movies_keyword_year_stats (mcv) +ON k.keyword, mk.year +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword_id = k.id); +ANALYZE join_mcv_movies; +SELECT pg_get_statisticsobjdef(oid, false) FROM pg_statistic_ext +WHERE stxname = 'join_mcv_movies_keyword_year_stats'; + pg_get_statisticsobjdef +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + join_mcv_keywords.keyword, join_mcv_movies.year FROM join_mcv_movies join_mcv_movies JOIN join_mcv_keywords join_mcv_keywords ON (join_mcv_movies.keyword_id = join_mcv_keywords.id) +(1 row) + +SELECT m.values, + m.nulls, + ROUND(m.frequency::numeric, 2) AS frequency +FROM pg_statistic_ext s +JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +CROSS JOIN LATERAL pg_mcv_list_items(d.stxdmcv) AS m +WHERE s.stxname = 'join_mcv_movies_keyword_year_stats' +ORDER BY m.frequency DESC, m.values; + values | nulls | frequency +-------------------+-------+----------- + {keyword_1,2020} | {f,f} | 0.06 + {keyword_10,2020} | {f,f} | 0.06 + {keyword_2,2020} | {f,f} | 0.06 + {keyword_3,2020} | {f,f} | 0.06 + {keyword_4,2020} | {f,f} | 0.06 + {keyword_5,2020} | {f,f} | 0.06 + {keyword_6,2020} | {f,f} | 0.06 + {keyword_7,2020} | {f,f} | 0.06 + {keyword_8,2020} | {f,f} | 0.06 + {keyword_9,2020} | {f,f} | 0.06 + {keyword_11,2021} | {f,f} | 0.03 + {keyword_12,2021} | {f,f} | 0.03 + {keyword_13,2021} | {f,f} | 0.03 + {keyword_14,2021} | {f,f} | 0.03 + {keyword_15,2021} | {f,f} | 0.03 + {keyword_16,2021} | {f,f} | 0.03 + {keyword_17,2021} | {f,f} | 0.03 + {keyword_18,2021} | {f,f} | 0.03 + {keyword_19,2021} | {f,f} | 0.03 + {keyword_20,2021} | {f,f} | 0.03 + {keyword_21,2022} | {f,f} | 0.01 + {keyword_22,2022} | {f,f} | 0.01 + {keyword_23,2022} | {f,f} | 0.01 + {keyword_24,2022} | {f,f} | 0.01 + {keyword_25,2022} | {f,f} | 0.01 + {keyword_26,2022} | {f,f} | 0.01 + {keyword_27,2022} | {f,f} | 0.01 + {keyword_28,2022} | {f,f} | 0.01 + {keyword_29,2022} | {f,f} | 0.01 + {keyword_30,2022} | {f,f} | 0.01 +(30 rows) + +-- keyword_1 always goes with 2020, which the statistics store +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_movies mk, join_mcv_keywords k + WHERE k.keyword = ''keyword_1'' AND mk.year = 2020 + AND k.id = mk.keyword_id +'); + estimated | actual +-----------+-------- + 600 | 600 +(1 row) + +-- keyword_1 never goes with 2021 +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_movies mk, join_mcv_keywords k + WHERE k.keyword = ''keyword_1'' AND mk.year = 2021 + AND k.id = mk.keyword_id +'); + estimated | actual +-----------+-------- + 1 | 0 +(1 row) + +-- The columns may be listed in either order +CREATE STATISTICS join_mcv_movies_year_keyword_stats (mcv) +ON mk.year, k.keyword +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword_id = k.id); +ANALYZE join_mcv_movies; +SELECT s.stxname, pg_get_statisticsobjdef(s.oid, false) +FROM pg_statistic_ext s +WHERE s.stxname = 'join_mcv_movies_year_keyword_stats'; + stxname | pg_get_statisticsobjdef +------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + join_mcv_movies_year_keyword_stats | join_mcv_movies.year, join_mcv_keywords.keyword FROM join_mcv_movies join_mcv_movies JOIN join_mcv_keywords join_mcv_keywords ON (join_mcv_movies.keyword_id = join_mcv_keywords.id) +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_movies mk, join_mcv_keywords k + WHERE k.keyword = ''keyword_1'' AND mk.year = 2020 + AND k.id = mk.keyword_id +'); + estimated | actual +-----------+-------- + 600 | 600 +(1 row) + +DROP STATISTICS join_mcv_movies_year_keyword_stats; +DROP STATISTICS join_mcv_movies_keyword_code_stats, + join_mcv_movies_keyword_year_stats; +-- Verify failures +CREATE STATISTICS join_mcv_bad_no_from (mcv) ON k.keyword; +ERROR: syntax error at or near ";" +LINE 1: CREATE STATISTICS join_mcv_bad_no_from (mcv) ON k.keyword; + ^ +-- a FROM list with a comma is not supported yet +CREATE STATISTICS join_mcv_bad_comma_list (mcv) ON k.keyword +FROM join_mcv_movies mk, join_mcv_keywords k; +ERROR: only a single relation or JOIN is allowed in CREATE STATISTICS +-- nor one with a join among its items +CREATE STATISTICS join_mcv_bad_comma (mcv) ON k.keyword +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword_id = k.id), + join_mcv_movies mk2; +ERROR: only a single relation or JOIN is allowed in CREATE STATISTICS +-- no columns +CREATE STATISTICS join_mcv_bad_no_columns (mcv) +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword_id = k.id); +ERROR: syntax error at or near "FROM" +LINE 2: FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyw... + ^ +-- an unqualified column name +CREATE STATISTICS join_mcv_bad_unqualified (mcv) ON keyword +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword_id = k.id); +ERROR: join statistics require table-qualified column names +-- an expression +CREATE STATISTICS join_mcv_bad_expression (mcv) ON lower(k.keyword) +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword_id = k.id); +ERROR: expressions are not supported in join statistics +-- other kinds than mcv +CREATE STATISTICS join_mcv_bad_ndistinct (ndistinct) ON k.keyword, mk.year +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword_id = k.id); +ERROR: only MCV statistics are supported for join statistics +HINT: ndistinct, dependencies, and expression statistics require a single table. +-- a condition on a single table would make the statistics describe only the +-- rows satisfying it, alone or with a join condition +CREATE STATISTICS join_mcv_bad_same_table (mcv) ON k.keyword +FROM join_mcv_movies mk + JOIN join_mcv_keywords k ON (mk.keyword_id = mk.movie_id); +ERROR: join statistics support only equality conditions between columns of two tables +LINE 3: JOIN join_mcv_keywords k ON (mk.keyword_id = mk.movie_i... + ^ +DETAIL: This condition does not refer to two tables. +CREATE STATISTICS join_mcv_bad_single_table_filter (mcv) ON k.keyword +FROM join_mcv_movies mk + JOIN join_mcv_keywords k ON (mk.keyword_id = k.id + AND mk.movie_id = mk.keyword_id); +ERROR: join statistics support only equality conditions between columns of two tables +LINE 4: AND mk.movie_id = mk.keywor... + ^ +DETAIL: This condition does not refer to two tables. +-- a non-equality join condition +CREATE STATISTICS join_mcv_bad_non_eq (mcv) ON k.keyword +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword_id > k.id); +ERROR: join statistics support only equality conditions between columns of two tables +LINE 2: ...M join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword... + ^ +DETAIL: This condition is not an equality between two columns. +-- an outer join +CREATE STATISTICS join_mcv_bad_left (mcv) ON k.keyword +FROM join_mcv_movies mk LEFT JOIN join_mcv_keywords k ON (mk.keyword_id = k.id); +ERROR: join statistics are only supported for inner joins +-- a whole-row reference in the join condition +CREATE STATISTICS join_mcv_bad_wholerow (mcv) ON k.keyword +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk = k); +ERROR: join statistics support only equality conditions between columns of two tables +LINE 2: ...ROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk = k); + ^ +DETAIL: This condition is not an equality between two columns. +-- a subquery in the FROM clause +CREATE STATISTICS join_mcv_bad_subquery (mcv) ON x.keyword +FROM (SELECT * FROM join_mcv_keywords) x + JOIN join_mcv_movies mk ON (x.id = mk.keyword_id); +ERROR: unsupported FROM clause element in join statistics +-- a sampled table +CREATE STATISTICS join_mcv_bad_tablesample (mcv) ON k.keyword +FROM join_mcv_movies mk TABLESAMPLE system (10) + JOIN join_mcv_keywords k ON (mk.keyword_id = k.id); +ERROR: unsupported FROM clause element in join statistics +-- a virtual generated column +CREATE TABLE join_mcv_vgc (id INTEGER PRIMARY KEY, v INTEGER, + g INTEGER GENERATED ALWAYS AS (v * 2) VIRTUAL); +CREATE STATISTICS join_mcv_bad_vgc (mcv) ON v.g +FROM join_mcv_movies mk JOIN join_mcv_vgc v ON (mk.keyword_id = v.id); +ERROR: statistics creation on virtual generated columns is not supported +DROP TABLE join_mcv_vgc; +-- a column of a type without a less-than operator, even alone, as the MCV +-- list is built by sorting +CREATE TABLE join_mcv_xid (id INTEGER PRIMARY KEY, w xid); +CREATE STATISTICS join_mcv_xid_stats (mcv) ON x.w +FROM join_mcv_movies mk JOIN join_mcv_xid x ON (mk.keyword_id = x.id); +ERROR: cannot create multivariate statistics on column "w" +DETAIL: The type xid has no default btree operator class. +CREATE STATISTICS join_mcv_xid_multi_stats (mcv) ON x.w, mk.keyword_id +FROM join_mcv_movies mk JOIN join_mcv_xid x ON (mk.keyword_id = x.id); +ERROR: cannot create multivariate statistics on column "w" +DETAIL: The type xid has no default btree operator class. +DROP TABLE join_mcv_xid; +-- a partitioned table, whichever of the tables it is +CREATE TABLE join_mcv_plain (id int); +CREATE TABLE join_mcv_partitioned (id int, val int) PARTITION BY RANGE (id); +CREATE TABLE join_mcv_partitioned_1 PARTITION OF join_mcv_partitioned + FOR VALUES FROM (0) TO (100); +CREATE TABLE join_mcv_partitioned_2 PARTITION OF join_mcv_partitioned + FOR VALUES FROM (100) TO (200); +CREATE INDEX ON join_mcv_partitioned (id); +CREATE STATISTICS join_mcv_part_probed_stats (mcv) ON p.val +FROM join_mcv_plain s JOIN join_mcv_partitioned p ON (s.id = p.id); +ERROR: cannot define statistics for relation "join_mcv_partitioned" +DETAIL: This operation is not supported for partitioned tables. +CREATE INDEX ON join_mcv_plain (id); +CREATE STATISTICS join_mcv_part_anchor_stats (mcv) ON s.id +FROM join_mcv_partitioned p JOIN join_mcv_plain s ON (p.id = s.id); +ERROR: cannot define statistics for relation "join_mcv_partitioned" +DETAIL: This operation is not supported for partitioned tables. +DROP TABLE join_mcv_plain, join_mcv_partitioned; +-- USING is another way of writing an equality join condition +CREATE TABLE join_mcv_keyword_notes (keyword_id INTEGER PRIMARY KEY, note TEXT) + WITH (autovacuum_enabled = off); +INSERT INTO join_mcv_keyword_notes SELECT id, 'note' FROM join_mcv_keywords; +ANALYZE join_mcv_keyword_notes; +CREATE STATISTICS join_mcv_using_stats (mcv) ON n.note +FROM join_mcv_movies mk JOIN join_mcv_keyword_notes n USING (keyword_id); +SELECT pg_get_statisticsobjdef(oid, false) FROM pg_statistic_ext +WHERE stxname = 'join_mcv_using_stats'; + pg_get_statisticsobjdef +----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + join_mcv_keyword_notes.note FROM join_mcv_movies join_mcv_movies JOIN join_mcv_keyword_notes join_mcv_keyword_notes ON (join_mcv_movies.keyword_id = join_mcv_keyword_notes.keyword_id) +(1 row) + +DROP STATISTICS join_mcv_using_stats; +-- a join of three tables is accepted, with a column from each joined table +-- printed with its own table and the tables stored in the order written; +-- ANALYZE warns that such statistics are not built yet +CREATE STATISTICS join_mcv_three_tables_stats (mcv) ON k.keyword, n.note + FROM join_mcv_movies mk + JOIN join_mcv_keywords k ON (mk.keyword_id = k.id) + JOIN join_mcv_keyword_notes n ON (k.id = n.keyword_id); +SELECT stxjoinrels::regclass[], pg_get_statisticsobjdef(oid, false) +FROM pg_statistic_ext WHERE stxname = 'join_mcv_three_tables_stats'; + stxjoinrels | pg_get_statisticsobjdef +------------------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + [0:2]={join_mcv_movies,join_mcv_keywords,join_mcv_keyword_notes} | join_mcv_keywords.keyword, join_mcv_keyword_notes.note FROM join_mcv_movies join_mcv_movies JOIN join_mcv_keywords join_mcv_keywords ON (join_mcv_movies.keyword_id = join_mcv_keywords.id) JOIN join_mcv_keyword_notes join_mcv_keyword_notes ON (join_mcv_keywords.id = join_mcv_keyword_notes.keyword_id) +(1 row) + +ANALYZE join_mcv_movies; +WARNING: skipping join statistics "public.join_mcv_three_tables_stats" +DETAIL: Statistics can currently be built only for joins of two tables on a single equality condition. +DROP STATISTICS join_mcv_three_tables_stats; +-- the same join parenthesized: the tables are stored in an order in which +-- each one is joined to an earlier one, here the notes before the keywords +CREATE STATISTICS join_mcv_nested_stats (mcv) ON k.keyword, n.note + FROM join_mcv_movies mk + JOIN (join_mcv_keywords k + JOIN join_mcv_keyword_notes n ON (k.id = n.keyword_id)) + ON (mk.keyword_id = n.keyword_id); +SELECT stxjoinrels::regclass[], pg_get_statisticsobjdef(oid, false) +FROM pg_statistic_ext WHERE stxname = 'join_mcv_nested_stats'; + stxjoinrels | pg_get_statisticsobjdef +------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + [0:2]={join_mcv_movies,join_mcv_keyword_notes,join_mcv_keywords} | join_mcv_keywords.keyword, join_mcv_keyword_notes.note FROM join_mcv_movies join_mcv_movies JOIN join_mcv_keyword_notes join_mcv_keyword_notes ON (join_mcv_movies.keyword_id = join_mcv_keyword_notes.keyword_id) JOIN join_mcv_keywords join_mcv_keywords ON (join_mcv_keywords.id = join_mcv_keyword_notes.keyword_id) +(1 row) + +DROP STATISTICS join_mcv_nested_stats; +-- a join condition may only refer to tables joined so far +CREATE STATISTICS join_mcv_bad_forward_ref (mcv) ON k.keyword, n.note + FROM join_mcv_movies mk + JOIN join_mcv_keywords k ON (mk.keyword_id = n.keyword_id) + JOIN join_mcv_keyword_notes n ON (k.id = n.keyword_id); +ERROR: missing FROM-clause entry for table "n" +LINE 3: JOIN join_mcv_keywords k ON (mk.keyword_id = n.keyword_i... + ^ +-- every table must be connected to the others by a join condition +CREATE STATISTICS join_mcv_bad_unconnected (mcv) ON k.keyword, n.note + FROM join_mcv_movies mk + JOIN join_mcv_keywords k ON (mk.keyword_id = k.id) + JOIN join_mcv_keyword_notes n ON (mk.movie_id = k.id); +ERROR: join conditions of join statistics must connect all the tables +DETAIL: No join condition connects "n" to "mk". +-- several conditions between the same two tables are accepted, but not +-- built yet either: ANALYZE skips them, and leaves statistics already in +-- place, such as restored ones +CREATE STATISTICS join_mcv_multiclause_stats (mcv) ON k.keyword +FROM join_mcv_movies mk + JOIN join_mcv_keywords k ON (mk.keyword_id = k.id AND mk.movie_id = k.id); +SELECT pg_get_statisticsobjdef(oid, false) FROM pg_statistic_ext +WHERE stxname = 'join_mcv_multiclause_stats'; + pg_get_statisticsobjdef +-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + join_mcv_keywords.keyword FROM join_mcv_movies join_mcv_movies JOIN join_mcv_keywords join_mcv_keywords ON (join_mcv_movies.keyword_id = join_mcv_keywords.id AND join_mcv_movies.movie_id = join_mcv_keywords.id) +(1 row) + +SELECT pg_restore_extended_stats( + 'schemaname', 'public', + 'relname', 'join_mcv_movies', + 'statistics_schemaname', 'public', + 'statistics_name', 'join_mcv_multiclause_stats', + 'inherited', false, + 'most_common_vals', '{{x}}'::text[], + 'most_common_freqs', '{1}'::double precision[], + 'most_common_base_freqs', '{1}'::double precision[]); + pg_restore_extended_stats +--------------------------- + t +(1 row) + +SELECT most_common_vals FROM pg_stats_ext +WHERE statistics_name = 'join_mcv_multiclause_stats'; + most_common_vals +------------------ + {{x}} +(1 row) + +ANALYZE join_mcv_movies; +WARNING: skipping join statistics "public.join_mcv_multiclause_stats" +DETAIL: Statistics can currently be built only for joins of two tables on a single equality condition. +SELECT most_common_vals FROM pg_stats_ext +WHERE statistics_name = 'join_mcv_multiclause_stats'; + most_common_vals +------------------ + {{x}} +(1 row) + +DROP STATISTICS join_mcv_multiclause_stats; +-- without a list of kinds, join statistics get every kind they support, +-- which is only mcv, whatever the number of columns +CREATE STATISTICS join_mcv_kinds_stats_1 ON k.keyword +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword_id = k.id); +CREATE STATISTICS join_mcv_kinds_stats_2 ON k.keyword, mk.year +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword_id = k.id); +SELECT stxname, stxkind FROM pg_statistic_ext +WHERE stxname LIKE 'join_mcv_kinds%' ORDER BY stxname; + stxname | stxkind +------------------------+--------- + join_mcv_kinds_stats_1 | {m} + join_mcv_kinds_stats_2 | {m} +(2 rows) + +DROP STATISTICS join_mcv_kinds_stats_1, join_mcv_kinds_stats_2; +-- check statistics are dropped when a column of theirs is, whichever table +-- it belongs to, and when a join column is +CREATE STATISTICS join_mcv_dep_note_stats (mcv) ON n.note +FROM join_mcv_movies mk JOIN join_mcv_keyword_notes n USING (keyword_id); +CREATE STATISTICS join_mcv_dep_year_stats (mcv) ON mk.year +FROM join_mcv_movies mk JOIN join_mcv_keyword_notes n USING (keyword_id); +ALTER TABLE join_mcv_keyword_notes DROP COLUMN note; +\d join_mcv_movies + Table "public.join_mcv_movies" + Column | Type | Collation | Nullable | Default +------------+---------+-----------+----------+--------- + movie_id | integer | | not null | + keyword_id | integer | | not null | + year | integer | | not null | +Indexes: + "join_mcv_movies_pkey" PRIMARY KEY, btree (movie_id) +Statistics objects: + "public.join_mcv_dep_year_stats" (mcv) ON join_mcv_movies.year FROM join_mcv_movies join_mcv_movies JOIN join_mcv_keyword_notes join_mcv_keyword_notes ON (join_mcv_movies.keyword_id = join_mcv_keyword_notes.keyword_id) + "public.join_mcv_movies_keyword_stats" (mcv) ON join_mcv_keywords.keyword FROM join_mcv_movies join_mcv_movies JOIN join_mcv_keywords join_mcv_keywords ON (join_mcv_movies.keyword_id = join_mcv_keywords.id) + +ALTER TABLE join_mcv_keyword_notes DROP COLUMN keyword_id; +\d join_mcv_movies + Table "public.join_mcv_movies" + Column | Type | Collation | Nullable | Default +------------+---------+-----------+----------+--------- + movie_id | integer | | not null | + keyword_id | integer | | not null | + year | integer | | not null | +Indexes: + "join_mcv_movies_pkey" PRIMARY KEY, btree (movie_id) +Statistics objects: + "public.join_mcv_movies_keyword_stats" (mcv) ON join_mcv_keywords.keyword FROM join_mcv_movies join_mcv_movies JOIN join_mcv_keywords join_mcv_keywords ON (join_mcv_movies.keyword_id = join_mcv_keywords.id) + +DROP TABLE join_mcv_keyword_notes; +-- check dropping a table in the middle of a join of 60 tables drops the +-- statistics (the statements are long, so they are not echoed) +\set ECHO none +SELECT stxname FROM pg_statistic_ext WHERE stxname = 'join_many_stats'; + stxname +----------------- + join_many_stats +(1 row) + +DROP TABLE join_many_30; +SELECT stxname FROM pg_statistic_ext WHERE stxname = 'join_many_stats'; + stxname +--------- +(0 rows) + +\set ECHO none +-- check the frequencies are fractions of the joined rows, scaled by the +-- size of the join: val = 'hit' is most of the probed table but few of the +-- joined rows, nearly all of which join id 10 +CREATE TABLE join_mcv_scale (id int) WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_scale_indexed (id int, val text) + WITH (autovacuum_enabled = off); +INSERT INTO join_mcv_scale +SELECT CASE WHEN g <= 9 THEN g ELSE 10 END FROM generate_series(1, 1000) g; +INSERT INTO join_mcv_scale_indexed +SELECT g, CASE WHEN g <= 9 THEN 'hit' ELSE 'miss' END +FROM generate_series(1, 10) g; +CREATE INDEX ON join_mcv_scale_indexed (id); +ANALYZE join_mcv_scale, join_mcv_scale_indexed; +-- Without join statistics: the filter and the join are treated as +-- independent +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_scale s + JOIN join_mcv_scale_indexed i ON (s.id = i.id) + WHERE i.val = ''hit'''); + estimated | actual +-----------+-------- + 900 | 9 +(1 row) + +-- With join statistics +CREATE STATISTICS join_mcv_scale_stats (mcv) ON i.val +FROM join_mcv_scale s JOIN join_mcv_scale_indexed i ON (s.id = i.id); +ANALYZE join_mcv_scale; +SELECT m.values, m.frequency, m.base_frequency +FROM pg_statistic_ext s +JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +CROSS JOIN LATERAL pg_mcv_list_items(d.stxdmcv) AS m +WHERE s.stxname = 'join_mcv_scale_stats' +ORDER BY m.frequency DESC; + values | frequency | base_frequency +--------+-----------+---------------- + {miss} | 0.991 | 0.991 + {hit} | 0.009 | 0.009 +(2 rows) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_scale s + JOIN join_mcv_scale_indexed i ON (s.id = i.id) + WHERE i.val = ''hit'''); + estimated | actual +-----------+-------- + 9 | 9 +(1 row) + +DROP TABLE join_mcv_scale, join_mcv_scale_indexed; +-- check statistics of two tables are used when a third table is joined to +-- either of them, in any order; the third table has one row per id, so it +-- changes neither the rows of the join nor the filter +CREATE TABLE join_mcv_three (id int) WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_three_indexed (id int, y int) + WITH (autovacuum_enabled = off); +CREATE INDEX ON join_mcv_three_indexed (id); +CREATE TABLE join_mcv_three_other (id int) WITH (autovacuum_enabled = off); +-- y = 1 in a few rows +INSERT INTO join_mcv_three_indexed +SELECT g, (g <= 10)::int FROM generate_series(1, 100) g; +-- most rows join the rows where y = 1 +INSERT INTO join_mcv_three +SELECT CASE WHEN g <= 900 THEN 1 + g % 10 ELSE 11 + g % 90 END +FROM generate_series(1, 1000) g; +-- 100 rows, one for each id +INSERT INTO join_mcv_three_other SELECT g FROM generate_series(1, 100) g; +ANALYZE join_mcv_three, join_mcv_three_indexed, join_mcv_three_other; +-- Without join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_three_other o + JOIN join_mcv_three f ON (o.id = f.id) + JOIN join_mcv_three_indexed i ON (f.id = i.id) + WHERE i.y = 1'); + estimated | actual +-----------+-------- + 100 | 900 +(1 row) + +CREATE STATISTICS join_mcv_three_stats (mcv) ON i.y +FROM join_mcv_three f JOIN join_mcv_three_indexed i ON (f.id = i.id); +ANALYZE join_mcv_three; +-- The two tables of the statistics alone +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_three f + JOIN join_mcv_three_indexed i ON (f.id = i.id) + WHERE i.y = 1'); + estimated | actual +-----------+-------- + 900 | 900 +(1 row) + +-- With the third table joined to either table of the statistics: the +-- estimate is the same whichever pair the planner joins first +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_three_other o + JOIN join_mcv_three f ON (o.id = f.id) + JOIN join_mcv_three_indexed i ON (f.id = i.id) + WHERE i.y = 1'); + estimated | actual +-----------+-------- + 900 | 900 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_three_other o + JOIN join_mcv_three_indexed i ON (o.id = i.id) + JOIN join_mcv_three f ON (f.id = i.id) + WHERE i.y = 1'); + estimated | actual +-----------+-------- + 900 | 900 +(1 row) + +DROP TABLE join_mcv_three, join_mcv_three_indexed, join_mcv_three_other; +-- check statistics correct a foreign key's estimate for a filter: the +-- foreign key does not account for NULL keys, and that error stays +CREATE TABLE join_mcv_fk_null_indexed (id int PRIMARY KEY, y int) + WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_fk_null (id int REFERENCES join_mcv_fk_null_indexed (id)) + WITH (autovacuum_enabled = off); +-- y = 1 in a few rows +INSERT INTO join_mcv_fk_null_indexed +SELECT g, (g <= 10)::int FROM generate_series(1, 100) g; +-- some rows reference no row, and most of the others reference the rows +-- where y = 1 +INSERT INTO join_mcv_fk_null +SELECT CASE WHEN g <= 200 THEN NULL + WHEN g <= 920 THEN 1 + g % 10 + ELSE 11 + g % 80 END +FROM generate_series(1, 1000) g; +ANALYZE join_mcv_fk_null, join_mcv_fk_null_indexed; +-- Without join statistics, with and without a filter +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_fk_null s + JOIN join_mcv_fk_null_indexed i ON (s.id = i.id)'); + estimated | actual +-----------+-------- + 1000 | 800 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_fk_null s + JOIN join_mcv_fk_null_indexed i ON (s.id = i.id) + WHERE i.y = 1'); + estimated | actual +-----------+-------- + 100 | 720 +(1 row) + +-- With join statistics, which correct the foreign key's estimate for the +-- filter only: the remaining error is the foreign key's own, as it does not +-- account for the NULL keys +CREATE STATISTICS join_mcv_fk_null_stats (mcv) ON i.y +FROM join_mcv_fk_null s JOIN join_mcv_fk_null_indexed i ON (s.id = i.id); +ANALYZE join_mcv_fk_null; +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_fk_null s + JOIN join_mcv_fk_null_indexed i ON (s.id = i.id) + WHERE i.y = 1'); + estimated | actual +-----------+-------- + 900 | 720 +(1 row) + +DROP TABLE join_mcv_fk_null, join_mcv_fk_null_indexed; +-- check estimates for values stored in the join statistics, and for values +-- not stored there +CREATE TABLE join_mcv_stored_indexed (id int PRIMARY KEY, b int, c int) + WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_stored (dim_id int, a int) + WITH (autovacuum_enabled = off); +-- common values of b, single rows, and a value no row joins +INSERT INTO join_mcv_stored_indexed +SELECT i, CASE WHEN i <= 30 THEN 1 WHEN i <= 50 THEN 2 WHEN i = 51 THEN 3 + WHEN i = 52 THEN 4 WHEN i <= 72 THEN 5 ELSE 6 END, + 1 + i % 4 +FROM generate_series(1, 80) i; +-- a is never below c +INSERT INTO join_mcv_stored +SELECT dim.id, dim.c + (g % 3 = 0)::int +FROM (VALUES (1, 30, 70), (31, 20, 50), (51, 1, 25), (52, 1, 15), (53, 20, 6)) + AS v (first_id, nrows, njoined), + generate_series(1, v.njoined) g, + join_mcv_stored_indexed dim +WHERE dim.id = v.first_id + g % v.nrows; +-- 10 rows that join nothing +INSERT INTO join_mcv_stored SELECT NULL, 1 FROM generate_series(1, 10); +ANALYZE join_mcv_stored_indexed, join_mcv_stored; +-- Without join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b = 1'); + estimated | actual +-----------+-------- + 62 | 70 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b IN (1, 3)'); + estimated | actual +-----------+-------- + 64 | 95 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b = 3'); + estimated | actual +-----------+-------- + 2 | 25 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b = 6'); + estimated | actual +-----------+-------- + 17 | 0 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b > 2'); + estimated | actual +-----------+-------- + 62 | 46 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE f.a = 2 AND dim.c = 2'); + estimated | actual +-----------+-------- + 9 | 22 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.c = 4'); + estimated | actual +-----------+-------- + 41 | 56 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE f.a > 2 AND dim.c > 2'); + estimated | actual +-----------+-------- + 47 | 89 +(1 row) + +-- With join statistics that store every value of b and every (a, c) +-- combination in the join +CREATE STATISTICS join_mcv_stored_b_stats (mcv) ON dim.b +FROM join_mcv_stored f JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id); +CREATE STATISTICS join_mcv_stored_a_c_stats (mcv) ON f.a, dim.c +FROM join_mcv_stored f JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id); +ANALYZE join_mcv_stored; +SELECT s.stxname, m.values, round(m.frequency::numeric, 3) AS frequency +FROM pg_statistic_ext s +JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +CROSS JOIN LATERAL pg_mcv_list_items(d.stxdmcv) AS m +WHERE s.stxname IN ('join_mcv_stored_b_stats', 'join_mcv_stored_a_c_stats') +ORDER BY s.stxname, m.frequency DESC; + stxname | values | frequency +---------------------------+--------+----------- + join_mcv_stored_a_c_stats | {4,4} | 0.229 + join_mcv_stored_a_c_stats | {1,1} | 0.169 + join_mcv_stored_a_c_stats | {3,3} | 0.145 + join_mcv_stored_a_c_stats | {2,2} | 0.133 + join_mcv_stored_a_c_stats | {5,4} | 0.108 + join_mcv_stored_a_c_stats | {2,1} | 0.102 + join_mcv_stored_a_c_stats | {3,2} | 0.060 + join_mcv_stored_a_c_stats | {4,3} | 0.054 + join_mcv_stored_b_stats | {1} | 0.422 + join_mcv_stored_b_stats | {2} | 0.301 + join_mcv_stored_b_stats | {3} | 0.151 + join_mcv_stored_b_stats | {4} | 0.090 + join_mcv_stored_b_stats | {5} | 0.036 +(13 rows) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b = 1'); + estimated | actual +-----------+-------- + 70 | 70 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b IN (1, 3)'); + estimated | actual +-----------+-------- + 95 | 95 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b = 3'); + estimated | actual +-----------+-------- + 25 | 25 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b = 6'); + estimated | actual +-----------+-------- + 1 | 0 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b > 2'); + estimated | actual +-----------+-------- + 46 | 46 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE f.a = 2 AND dim.c = 2'); + estimated | actual +-----------+-------- + 22 | 22 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.c = 4'); + estimated | actual +-----------+-------- + 56 | 56 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE f.a > 2 AND dim.c > 2'); + estimated | actual +-----------+-------- + 89 | 89 +(1 row) + +-- With lower statistics targets, only the most common values are stored; +-- the joined rows all fit in the sample, so the statistics are the same on +-- every run +ALTER STATISTICS join_mcv_stored_b_stats SET STATISTICS 2; +ALTER STATISTICS join_mcv_stored_a_c_stats SET STATISTICS 3; +ANALYZE join_mcv_stored; +SELECT s.stxname, m.values, round(m.frequency::numeric, 3) AS frequency +FROM pg_statistic_ext s +JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +CROSS JOIN LATERAL pg_mcv_list_items(d.stxdmcv) AS m +WHERE s.stxname IN ('join_mcv_stored_b_stats', 'join_mcv_stored_a_c_stats') +ORDER BY s.stxname, m.frequency DESC; + stxname | values | frequency +---------------------------+--------+----------- + join_mcv_stored_a_c_stats | {4,4} | 0.229 + join_mcv_stored_a_c_stats | {1,1} | 0.169 + join_mcv_stored_a_c_stats | {3,3} | 0.145 + join_mcv_stored_b_stats | {1} | 0.422 + join_mcv_stored_b_stats | {2} | 0.301 +(5 rows) + +-- A stored value. +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b = 1'); + estimated | actual +-----------+-------- + 70 | 70 +(1 row) + +-- One value stored and one not: the frequency of b = 1, plus the estimate +-- without join statistics for b = 3 +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b IN (1, 3)'); + estimated | actual +-----------+-------- + 72 | 95 +(1 row) + +-- A value not stored gets the estimate without join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b = 3'); + estimated | actual +-----------+-------- + 2 | 25 +(1 row) + +-- The statistics no longer show that b = 6 joins nothing, so the estimate +-- without join statistics is kept +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b = 6'); + estimated | actual +-----------+-------- + 17 | 0 +(1 row) + +-- A range no stored value satisfies: the estimate is capped at the joined +-- rows of the values not stored +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b > 2'); + estimated | actual +-----------+-------- + 46 | 46 +(1 row) + +-- A combination not stored gets the estimate without join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE f.a = 2 AND dim.c = 2'); + estimated | actual +-----------+-------- + 9 | 22 +(1 row) + +-- (4, 4) is stored; the other combinations with c = 4 get the estimate +-- without join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.c = 4'); + estimated | actual +-----------+-------- + 68 | 56 +(1 row) + +-- A range: (3, 3) and (4, 4) are stored, and the other combinations it +-- selects get the estimate without join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE f.a > 2 AND dim.c > 2'); + estimated | actual +-----------+-------- + 90 | 89 +(1 row) + +-- Leave join_mcv_stored, join_mcv_stored_indexed and their statistics in +-- place for pg_upgrade testing. +-- check a value stored at a frequency much lower in the join than in its +-- table: b = 0 is 30 percent of the dimension rows but 1 percent of the +-- joined rows, the most common value there +CREATE TABLE join_mcv_diluted (dim_id int) WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_diluted_indexed (id int PRIMARY KEY, b int) + WITH (autovacuum_enabled = off); +-- b = 0 in many rows, and a distinct b in each other row +INSERT INTO join_mcv_diluted_indexed +SELECT g, CASE WHEN g <= 90 THEN 0 ELSE g END FROM generate_series(1, 300) g; +-- few rows reference the rows where b = 0; the others reference more values +-- of b than the statistics store +INSERT INTO join_mcv_diluted +SELECT CASE WHEN g <= 10 THEN g ELSE 91 + g % 210 END +FROM generate_series(1, 1000) g; +ANALYZE join_mcv_diluted, join_mcv_diluted_indexed; +-- Without join statistics: b = 0 is 30 percent of the dimension table +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_diluted f + JOIN join_mcv_diluted_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b = 0'); + estimated | actual +-----------+-------- + 300 | 10 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_diluted f + JOIN join_mcv_diluted_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b < 1'); + estimated | actual +-----------+-------- + 300 | 10 +(1 row) + +-- With join statistics, which store b = 0 at 1 percent of the joined rows +CREATE STATISTICS join_mcv_diluted_stats (mcv) ON dim.b +FROM join_mcv_diluted f + JOIN join_mcv_diluted_indexed dim ON (f.dim_id = dim.id); +ANALYZE join_mcv_diluted; +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_diluted f + JOIN join_mcv_diluted_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b = 0'); + estimated | actual +-----------+-------- + 10 | 10 +(1 row) + +-- A range selecting that value alone is measured the same way +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_diluted f + JOIN join_mcv_diluted_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b < 1'); + estimated | actual +-----------+-------- + 10 | 10 +(1 row) + +DROP TABLE join_mcv_diluted, join_mcv_diluted_indexed; +-- check a filter on the anchor table expected to select less than one row +-- is corrected as selecting one row, as a filter on the probed table is, +-- since the planner estimates at least one row for a table +CREATE TABLE join_mcv_rare (id int, a int) WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_rare_indexed (id int) WITH (autovacuum_enabled = off); +CREATE INDEX ON join_mcv_rare_indexed (id); +-- half the rows have id 1 +INSERT INTO join_mcv_rare_indexed +SELECT CASE WHEN g <= 500 THEN 1 ELSE g - 499 END +FROM generate_series(1, 1000) g; +-- the one row with a = -10 references id 1; the others, with a = 0, +-- reference one row each +INSERT INTO join_mcv_rare +SELECT CASE WHEN g = 1 THEN 1 ELSE 2 + g % 500 END, + CASE WHEN g = 1 THEN -10 ELSE 0 END +FROM generate_series(1, 1000) g; +ANALYZE join_mcv_rare, join_mcv_rare_indexed; +-- Without join statistics: a = -10, and a < -5 with no histogram to +-- estimate it from, are each expected to select one row +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_rare s + JOIN join_mcv_rare_indexed i ON (s.id = i.id) + WHERE s.a = -10'); + estimated | actual +-----------+-------- + 2 | 500 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_rare s + JOIN join_mcv_rare_indexed i ON (s.id = i.id) + WHERE s.a < -5'); + estimated | actual +-----------+-------- + 2 | 500 +(1 row) + +-- With join statistics, which store that a third of the joined rows have +-- a = -10: the two filters get the same estimate +CREATE STATISTICS join_mcv_rare_stats (mcv) ON s.a +FROM join_mcv_rare s JOIN join_mcv_rare_indexed i ON (s.id = i.id); +ANALYZE join_mcv_rare; +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_rare s + JOIN join_mcv_rare_indexed i ON (s.id = i.id) + WHERE s.a = -10'); + estimated | actual +-----------+-------- + 599 | 500 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_rare s + JOIN join_mcv_rare_indexed i ON (s.id = i.id) + WHERE s.a < -5'); + estimated | actual +-----------+-------- + 599 | 500 +(1 row) + +DROP TABLE join_mcv_rare, join_mcv_rare_indexed; +-- check which filter clauses the join statistics can estimate +CREATE TABLE join_mcv_filters_indexed (id int PRIMARY KEY, b int, c int, d bool) + WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_filters (dim_id int, a int) + WITH (autovacuum_enabled = off); +-- c NULL in some rows, d true in fewer +INSERT INTO join_mcv_filters_indexed +SELECT i, 1 + i % 4, NULLIF(i % 5, 0), i % 10 = 0 +FROM generate_series(1, 40) i; +-- a third of the rows join the rows where c is NULL; a is never below b +INSERT INTO join_mcv_filters +SELECT dim.id, dim.b + (g % 3 = 0)::int +FROM generate_series(1, 600) g +JOIN join_mcv_filters_indexed dim + ON dim.id = CASE WHEN g <= 200 THEN 5 * (1 + g % 8) ELSE 1 + g % 40 END; +ANALYZE join_mcv_filters_indexed, join_mcv_filters; +-- Without join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE f.a > 2 AND dim.b > 2'); + estimated | actual +-----------+-------- + 175 | 300 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE 2 < f.a AND 2 < dim.b'); + estimated | actual +-----------+-------- + 175 | 300 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE f.a <> 1 AND dim.b <> 1'); + estimated | actual +-----------+-------- + 375 | 450 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE f.a IN (1, 2) AND dim.b = 1'); + estimated | actual +-----------+-------- + 62 | 150 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE f.a IN (1, NULL, 2) AND dim.b = 1'); + estimated | actual +-----------+-------- + 62 | 150 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE f.a <= ANY (ARRAY[1, 2]) AND dim.b = 1'); + estimated | actual +-----------+-------- + 77 | 150 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE f.a < ALL (ARRAY[3, 4]) AND dim.b = 2'); + estimated | actual +-----------+-------- + 42 | 100 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE dim.c NOT IN (1, 2)'); + estimated | actual +-----------+-------- + 120 | 160 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE (f.a = 1 OR f.a = 5) AND dim.b = 4'); + estimated | actual +-----------+-------- + 36 | 50 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE (dim.c = 1 OR dim.c IS NULL)'); + estimated | actual +-----------+-------- + 210 | 360 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE dim.c IS NULL'); + estimated | actual +-----------+-------- + 120 | 280 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE dim.c IS NOT NULL'); + estimated | actual +-----------+-------- + 480 | 320 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE dim.d'); + estimated | actual +-----------+-------- + 60 | 140 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE NOT dim.d'); + estimated | actual +-----------+-------- + 540 | 460 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE dim.d IS TRUE'); + estimated | actual +-----------+-------- + 60 | 140 +(1 row) + +-- With join statistics that store every combination in the join +CREATE STATISTICS join_mcv_filters_stats (mcv) ON f.a, dim.b, dim.c, dim.d +FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id); +ANALYZE join_mcv_filters; +SELECT count(*) AS items, round(sum(m.frequency)::numeric, 2) AS total_frequency +FROM pg_statistic_ext s +JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +CROSS JOIN LATERAL pg_mcv_list_items(d.stxdmcv) AS m +WHERE s.stxname = 'join_mcv_filters_stats'; + items | total_frequency +-------+----------------- + 40 | 1.00 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE f.a > 2 AND dim.b > 2'); + estimated | actual +-----------+-------- + 300 | 300 +(1 row) + +-- A constant on the left: the same as "f.a > 2 AND dim.b > 2". +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE 2 < f.a AND 2 < dim.b'); + estimated | actual +-----------+-------- + 300 | 300 +(1 row) + +-- "<>" on both columns. +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE f.a <> 1 AND dim.b <> 1'); + estimated | actual +-----------+-------- + 450 | 450 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE f.a IN (1, 2) AND dim.b = 1'); + estimated | actual +-----------+-------- + 150 | 150 +(1 row) + +-- NULL array elements are dropped, so this is the same as "f.a IN (1, 2)". +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE f.a IN (1, NULL, 2) AND dim.b = 1'); + estimated | actual +-----------+-------- + 150 | 150 +(1 row) + +-- An inequality over an array: the same as "f.a IN (1, 2)" here. +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE f.a <= ANY (ARRAY[1, 2]) AND dim.b = 1'); + estimated | actual +-----------+-------- + 150 | 150 +(1 row) + +-- ALL: the same as "f.a < 3". +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE f.a < ALL (ARRAY[3, 4]) AND dim.b = 2'); + estimated | actual +-----------+-------- + 100 | 100 +(1 row) + +-- NOT IN is "<> ALL". +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE dim.c NOT IN (1, 2)'); + estimated | actual +-----------+-------- + 160 | 160 +(1 row) + +-- OR of filters on one table. +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE (f.a = 1 OR f.a = 5) AND dim.b = 4'); + estimated | actual +-----------+-------- + 50 | 50 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE (dim.c = 1 OR dim.c IS NULL)'); + estimated | actual +-----------+-------- + 350 | 360 +(1 row) + +-- IS [NOT] NULL. +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE dim.c IS NULL'); + estimated | actual +-----------+-------- + 280 | 280 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE dim.c IS NOT NULL'); + estimated | actual +-----------+-------- + 320 | 320 +(1 row) + +-- A boolean column, alone and negated. +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE dim.d'); + estimated | actual +-----------+-------- + 140 | 140 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE NOT dim.d'); + estimated | actual +-----------+-------- + 460 | 460 +(1 row) + +-- IS TRUE is not a form the statistics match: estimated as without them. +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE dim.d IS TRUE'); + estimated | actual +-----------+-------- + 60 | 140 +(1 row) + +-- Leave join_mcv_filters, join_mcv_filters_indexed and their statistics in +-- place for pg_upgrade testing. +-- check a filter the join statistics do not cover is estimated from the +-- probed table's own statistics: color and shape always go together, and +-- the join statistics cover color only +CREATE TABLE join_mcv_partial_indexed ( + id INTEGER PRIMARY KEY, + color TEXT NOT NULL, + shape TEXT NOT NULL +) WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_partial ( + id INTEGER PRIMARY KEY, + dim_id INTEGER NOT NULL +) WITH (autovacuum_enabled = off); +-- each color with its own shape +INSERT INTO join_mcv_partial_indexed (id, color, shape) +SELECT i, + CASE (i - 1) / 20 + WHEN 0 THEN 'red' + WHEN 1 THEN 'blue' + WHEN 2 THEN 'green' + WHEN 3 THEN 'yellow' + ELSE 'white' + END, + CASE (i - 1) / 20 + WHEN 0 THEN 'circle' + WHEN 1 THEN 'square' + WHEN 2 THEN 'triangle' + WHEN 3 THEN 'star' + ELSE 'diamond' + END +FROM generate_series(1, 100) i; +-- half the rows reference the red rows, none the yellow or white ones +INSERT INTO join_mcv_partial (id, dim_id) +SELECT i, + CASE + WHEN i % 100 < 50 THEN (i % 20) + 1 + WHEN i % 100 < 80 THEN (i % 20) + 21 + ELSE (i % 20) + 41 + END +FROM generate_series(1, 10000) i; +ANALYZE join_mcv_partial_indexed; +ANALYZE join_mcv_partial; +-- Single-table statistics on the probed table know that color and shape go +-- together +CREATE STATISTICS join_mcv_partial_single_stats (mcv) ON color, shape +FROM join_mcv_partial_indexed; +ANALYZE join_mcv_partial_indexed; +-- Without join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_partial f, join_mcv_partial_indexed d + WHERE d.color = ''red'' AND d.shape = ''circle'' + AND f.dim_id = d.id +'); + estimated | actual +-----------+-------- + 2000 | 5000 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_partial f, join_mcv_partial_indexed d + WHERE d.color = ''red'' AND d.shape = ''square'' + AND f.dim_id = d.id +'); + estimated | actual +-----------+-------- + 100 | 0 +(1 row) + +-- With join statistics covering color only +CREATE STATISTICS join_mcv_partial_stats (mcv) ON d.color +FROM join_mcv_partial f JOIN join_mcv_partial_indexed d ON (f.dim_id = d.id); +ANALYZE join_mcv_partial; +-- red and circle go together: the statistics measure the pair +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_partial f, join_mcv_partial_indexed d + WHERE d.color = ''red'' AND d.shape = ''circle'' + AND f.dim_id = d.id +'); + estimated | actual +-----------+-------- + 5000 | 5000 +(1 row) + +-- red and square never go together: the estimate is the one without join +-- statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_partial f, join_mcv_partial_indexed d + WHERE d.color = ''red'' AND d.shape = ''square'' + AND f.dim_id = d.id +'); + estimated | actual +-----------+-------- + 100 | 0 +(1 row) + +DROP TABLE join_mcv_partial, join_mcv_partial_indexed CASCADE; +-- check two statistics objects are used in one join, one for each +-- dimension table of a fact table +CREATE TABLE join_mcv_star ( + id INTEGER PRIMARY KEY, + dim1_id INTEGER NOT NULL, + dim2_id INTEGER NOT NULL +) WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_star_dim1 ( + id INTEGER PRIMARY KEY, + label TEXT NOT NULL +) WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_star_dim2 ( + id INTEGER PRIMARY KEY, + color TEXT NOT NULL +) WITH (autovacuum_enabled = off); +-- join_mcv_star_dim1: 50 rows +INSERT INTO join_mcv_star_dim1 (id, label) +SELECT i, 'label_' || i FROM generate_series(1, 50) i; +-- join_mcv_star_dim2: 20 rows +INSERT INTO join_mcv_star_dim2 (id, color) +SELECT i, 'color_' || i FROM generate_series(1, 20) i; +-- skewed to both dimensions: dim1_id mostly among the first ten ids, +-- dim2_id mostly id 1, and some rows reference no dim2 row +INSERT INTO join_mcv_star (id, dim1_id, dim2_id) +SELECT i, + CASE + WHEN i % 100 < 60 THEN (i % 10) + 1 + WHEN i % 100 < 90 THEN (i % 10) + 11 + ELSE (i % 10) + 21 + END, + CASE + WHEN i % 100 < 50 THEN 1 + WHEN i % 100 < 80 THEN 2 + ELSE (i % 20) + 3 + END +FROM generate_series(1, 10000) i; +ANALYZE join_mcv_star; +ANALYZE join_mcv_star_dim1; +ANALYZE join_mcv_star_dim2; +-- Without join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_star f + JOIN join_mcv_star_dim1 d1 ON (f.dim1_id = d1.id) + JOIN join_mcv_star_dim2 d2 ON (f.dim2_id = d2.id) + WHERE d1.label = ''label_1'' AND d2.color = ''color_1'' +'); + estimated | actual +-----------+-------- + 9 | 500 +(1 row) + +-- With join statistics for each pair +CREATE STATISTICS join_mcv_star_dim1_stats (mcv) ON d1.label +FROM join_mcv_star f JOIN join_mcv_star_dim1 d1 ON (f.dim1_id = d1.id); +ANALYZE join_mcv_star; +CREATE STATISTICS join_mcv_star_dim2_stats (mcv) ON d2.color +FROM join_mcv_star f JOIN join_mcv_star_dim2 d2 ON (f.dim2_id = d2.id); +ANALYZE join_mcv_star; +-- Three-way join with both objects applicable: label_1 and color_1 go +-- together, which neither object knows, so the two corrections combine as +-- independent +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_star f + JOIN join_mcv_star_dim1 d1 ON (f.dim1_id = d1.id) + JOIN join_mcv_star_dim2 d2 ON (f.dim2_id = d2.id) + WHERE d1.label = ''label_1'' AND d2.color = ''color_1'' +'); + estimated | actual +-----------+-------- + 278 | 500 +(1 row) + +-- Dropping the color object: label_1 is still measured, color_1 is +-- estimated as 1 color in 20 +DROP STATISTICS join_mcv_star_dim2_stats; +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_star f + JOIN join_mcv_star_dim1 d1 ON (f.dim1_id = d1.id) + JOIN join_mcv_star_dim2 d2 ON (f.dim2_id = d2.id) + WHERE d1.label = ''label_1'' AND d2.color = ''color_1'' +'); + estimated | actual +-----------+-------- + 27 | 500 +(1 row) + +-- No statistics at all +DROP STATISTICS join_mcv_star_dim1_stats; +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_star f + JOIN join_mcv_star_dim1 d1 ON (f.dim1_id = d1.id) + JOIN join_mcv_star_dim2 d2 ON (f.dim2_id = d2.id) + WHERE d1.label = ''label_1'' AND d2.color = ''color_1'' +'); + estimated | actual +-----------+-------- + 9 | 500 +(1 row) + +DROP TABLE join_mcv_star, join_mcv_star_dim1, join_mcv_star_dim2; +-- check an inequality join is not the condition the statistics were built +-- on, so the estimate is the one without statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_movies mk, join_mcv_keywords k + WHERE k.keyword = ''keyword_1'' AND mk.keyword_id > k.id +'); + estimated | actual +-----------+-------- + 3333 | 9400 +(1 row) + +DROP STATISTICS join_mcv_movies_keyword_stats; +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_movies mk, join_mcv_keywords k + WHERE k.keyword = ''keyword_1'' AND mk.keyword_id > k.id +'); + estimated | actual +-----------+-------- + 3333 | 9400 +(1 row) + +-- check join statistics are not used for a join of their tables on more +-- than one condition, and what counts as a condition +CREATE TABLE join_mcv_two (a int, b int) WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_two_indexed (a int, b int, y int) + WITH (autovacuum_enabled = off); +CREATE INDEX ON join_mcv_two_indexed (a); +CREATE INDEX ON join_mcv_two_indexed (b); +-- every pair (a, b) once, with y = 1 where a or b is 1 +INSERT INTO join_mcv_two_indexed +SELECT a, b, (a = 1 OR b = 1)::int +FROM generate_series(1, 10) a, generate_series(1, 10) b; +-- most rows join (1, 1), where y = 1 +INSERT INTO join_mcv_two +SELECT CASE WHEN g <= 900 THEN 1 ELSE 2 + g % 9 END, + CASE WHEN g <= 900 THEN 1 ELSE 2 + (g / 9) % 9 END +FROM generate_series(1, 1000) g; +ANALYZE join_mcv_two, join_mcv_two_indexed; +-- Without join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_two s + JOIN join_mcv_two_indexed i ON (s.a = i.a AND s.b = i.b) + WHERE i.y = 1'); + estimated | actual +-----------+-------- + 190 | 900 +(1 row) + +-- With join statistics for one of the conditions +CREATE STATISTICS join_mcv_two_a_stats (mcv) ON i.y +FROM join_mcv_two s JOIN join_mcv_two_indexed i ON (s.a = i.a); +ANALYZE join_mcv_two; +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_two s + JOIN join_mcv_two_indexed i ON (s.a = i.a AND s.b = i.b) + WHERE i.y = 1'); + estimated | actual +-----------+-------- + 190 | 900 +(1 row) + +-- A condition equal to a constant becomes a restriction on each table rather +-- than a join condition, so the join statistics are used +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_two s + JOIN join_mcv_two_indexed i ON (s.a = i.a AND s.b = i.b) + WHERE i.y = 1 AND i.b = 1'); + estimated | actual +-----------+-------- + 862 | 900 +(1 row) + +-- Expressions over both tables, with join statistics for one of the +-- conditions. An expression that is only sorted on is not a join +-- condition, so the join statistics are used +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_two s + JOIN join_mcv_two_indexed i ON (s.a = i.a) + WHERE i.y = 1 + ORDER BY s.b + i.b'); + estimated | actual +-----------+-------- + 9100 | 9100 +(1 row) + +-- An expression equal to the join key is a second join condition, so the +-- join statistics are not used +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_two s + JOIN join_mcv_two_indexed i ON (s.a = i.a AND i.a = s.b + i.b - 1) + WHERE i.y = 1'); + estimated | actual +-----------+-------- + 9 | 911 +(1 row) + +-- Outer joins, with join statistics for one of the conditions +CREATE TABLE join_mcv_third (k int) WITH (autovacuum_enabled = off); +INSERT INTO join_mcv_third SELECT g FROM generate_series(1, 1000) g; +ANALYZE join_mcv_third; +-- A filter on the anchor table in the join condition of a left join is not a +-- join condition, so the join statistics are used +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_two s + LEFT JOIN join_mcv_two_indexed i ON (s.a = i.a AND i.y = 1 AND s.b = 1)'); + estimated | actual +-----------+-------- + 8190 | 9100 +(1 row) + +-- A condition on both tables above a left join is a second join condition, +-- so the join statistics are not used +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_two s + LEFT JOIN join_mcv_two_indexed i ON (s.a = i.a AND i.y = 1) + WHERE (i.b = s.b OR i.b IS NULL)'); + estimated | actual +-----------+-------- + 190 | 900 +(1 row) + +-- One condition, on a column nullable by a full join below, so the join +-- statistics are used +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_third t + FULL JOIN join_mcv_two s ON (t.k = s.a) + LEFT JOIN join_mcv_two_indexed i ON (s.a = i.a AND i.y = 1)'); + estimated | actual +-----------+-------- + 9100 | 10090 +(1 row) + +-- The same with a left join below, for which the planner keeps a copy of +-- the join clause for each join order +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_third t + LEFT JOIN join_mcv_two s ON (t.k = s.a) + LEFT JOIN join_mcv_two_indexed i ON (s.a = i.a AND i.y = 1)'); + estimated | actual +-----------+-------- + 9100 | 10090 +(1 row) + +DROP TABLE join_mcv_two, join_mcv_two_indexed, join_mcv_third; +-- check a foreign key with another condition counts as two conditions +CREATE TABLE join_mcv_fk_indexed (id int PRIMARY KEY, b int, y int) + WITH (autovacuum_enabled = off); +CREATE INDEX ON join_mcv_fk_indexed (b); +CREATE TABLE join_mcv_fk (id int REFERENCES join_mcv_fk_indexed (id), b int) + WITH (autovacuum_enabled = off); +-- groups of ten rows by b, with y = 1 only in the group b = 0 +INSERT INTO join_mcv_fk_indexed +SELECT g, (g - 1) / 10, (g <= 10)::int FROM generate_series(1, 200) g; +-- few rows have the b of the row they reference +INSERT INTO join_mcv_fk +SELECT g % 10 + 1, (g / 10) % 20 FROM generate_series(0, 999) g; +INSERT INTO join_mcv_fk +SELECT i.id, CASE WHEN g % 20 = 0 THEN i.b ELSE 0 END +FROM generate_series(0, 999) g +JOIN join_mcv_fk_indexed i ON (i.id = 11 + g % 190); +ANALYZE join_mcv_fk, join_mcv_fk_indexed; +-- Without join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_fk s + JOIN join_mcv_fk_indexed i ON (s.id = i.id AND s.b = i.b) + WHERE i.y = 1'); + estimated | actual +-----------+-------- + 5 | 50 +(1 row) + +-- With join statistics for the foreign key: the estimate is unchanged +CREATE STATISTICS join_mcv_fk_id_stats (mcv) ON i.y +FROM join_mcv_fk s JOIN join_mcv_fk_indexed i ON (s.id = i.id); +ANALYZE join_mcv_fk; +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_fk s + JOIN join_mcv_fk_indexed i ON (s.id = i.id AND s.b = i.b) + WHERE i.y = 1'); + estimated | actual +-----------+-------- + 5 | 50 +(1 row) + +DROP TABLE join_mcv_fk, join_mcv_fk_indexed; +-- check two foreign keys count as two conditions, and the error is in the +-- planner's estimate of the join itself, which no correction can repair +CREATE TABLE join_mcv_fk2_indexed (id int PRIMARY KEY, code int UNIQUE, y int) + WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_fk2 (id int REFERENCES join_mcv_fk2_indexed (id), + code int REFERENCES join_mcv_fk2_indexed (code)) + WITH (autovacuum_enabled = off); +-- code = id, with y = 1 in a few rows +INSERT INTO join_mcv_fk2_indexed +SELECT g, g, (g <= 10)::int FROM generate_series(1, 100) g; +-- half the rows reference the rows where y = 1, by both keys +INSERT INTO join_mcv_fk2 +SELECT r, r +FROM (SELECT CASE WHEN g <= 500 THEN 1 + g % 10 ELSE 11 + g % 90 END + FROM generate_series(1, 1000) g) AS s(r); +ANALYZE join_mcv_fk2, join_mcv_fk2_indexed; +-- Without join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_fk2 s + JOIN join_mcv_fk2_indexed i ON (s.id = i.id AND s.code = i.code) + WHERE i.y = 1'); + estimated | actual +-----------+-------- + 1 | 500 +(1 row) + +-- With join statistics for each of the foreign keys: neither applies +CREATE STATISTICS join_mcv_fk2_id_stats (mcv) ON i.y +FROM join_mcv_fk2 s JOIN join_mcv_fk2_indexed i ON (s.id = i.id); +CREATE STATISTICS join_mcv_fk2_code_stats (mcv) ON i.y +FROM join_mcv_fk2 s JOIN join_mcv_fk2_indexed i ON (s.code = i.code); +ANALYZE join_mcv_fk2; +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_fk2 s + JOIN join_mcv_fk2_indexed i ON (s.id = i.id AND s.code = i.code) + WHERE i.y = 1'); + estimated | actual +-----------+-------- + 1 | 500 +(1 row) + +DROP TABLE join_mcv_fk2, join_mcv_fk2_indexed; +-- check semi and anti joins are not corrected, and EXISTS on a primary +-- key, which becomes an inner join, is +CREATE TABLE join_mcv_semi_indexed (id int PRIMARY KEY, b int) + WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_semi (dim_id int) WITH (autovacuum_enabled = off); +-- b = 1 in a few rows +INSERT INTO join_mcv_semi_indexed +SELECT i, (i <= 10)::int FROM generate_series(1, 100) i; +-- most rows join the rows where b = 1; the last rows are joined by none +INSERT INTO join_mcv_semi +SELECT CASE WHEN g <= 900 THEN 1 + g % 10 ELSE 11 + g % 50 END +FROM generate_series(1, 1000) g; +ANALYZE join_mcv_semi_indexed, join_mcv_semi; +-- Without join statistics +-- An inner join, for comparison. +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_semi f + JOIN join_mcv_semi_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b = 1'); + estimated | actual +-----------+-------- + 100 | 900 +(1 row) + +-- Rows of join_mcv_semi with a match where b = 1, and without one. +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_semi f + WHERE EXISTS (SELECT 1 FROM join_mcv_semi_indexed dim + WHERE dim.id = f.dim_id AND dim.b = 1)'); + estimated | actual +-----------+-------- + 100 | 900 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_semi f + WHERE NOT EXISTS (SELECT 1 FROM join_mcv_semi_indexed dim + WHERE dim.id = f.dim_id AND dim.b = 1)'); + estimated | actual +-----------+-------- + 900 | 100 +(1 row) + +-- Rows of join_mcv_semi_indexed with b = 0 that are joined, and that are not. +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_semi_indexed dim + WHERE dim.b = 0 + AND EXISTS (SELECT 1 FROM join_mcv_semi f + WHERE f.dim_id = dim.id)'); + estimated | actual +-----------+-------- + 54 | 50 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_semi_indexed dim + WHERE dim.b = 0 + AND NOT EXISTS (SELECT 1 FROM join_mcv_semi f + WHERE f.dim_id = dim.id)'); + estimated | actual +-----------+-------- + 36 | 40 +(1 row) + +-- With join statistics that store every value of b in the join +CREATE STATISTICS join_mcv_semi_stats (mcv) ON dim.b +FROM join_mcv_semi f JOIN join_mcv_semi_indexed dim ON (f.dim_id = dim.id); +ANALYZE join_mcv_semi; +-- An inner join, for comparison. +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_semi f + JOIN join_mcv_semi_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b = 1'); + estimated | actual +-----------+-------- + 900 | 900 +(1 row) + +-- Rows of join_mcv_semi with a match where b = 1: EXISTS on the primary key +-- of join_mcv_semi_indexed becomes an inner join, which the statistics correct +-- as above +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_semi f + WHERE EXISTS (SELECT 1 FROM join_mcv_semi_indexed dim + WHERE dim.id = f.dim_id AND dim.b = 1)'); + estimated | actual +-----------+-------- + 900 | 900 +(1 row) + +-- and without one: an anti join keeps its estimate, as the statistics do not +-- record which rows have no match +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_semi f + WHERE NOT EXISTS (SELECT 1 FROM join_mcv_semi_indexed dim + WHERE dim.id = f.dim_id AND dim.b = 1)'); + estimated | actual +-----------+-------- + 900 | 100 +(1 row) + +-- Rows of join_mcv_semi_indexed with b = 0 that are joined, and that are not: +-- with join_mcv_semi_indexed on the outer side the statistics are not used +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_semi_indexed dim + WHERE dim.b = 0 + AND EXISTS (SELECT 1 FROM join_mcv_semi f + WHERE f.dim_id = dim.id)'); + estimated | actual +-----------+-------- + 54 | 50 +(1 row) + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_semi_indexed dim + WHERE dim.b = 0 + AND NOT EXISTS (SELECT 1 FROM join_mcv_semi f + WHERE f.dim_id = dim.id)'); + estimated | actual +-----------+-------- + 36 | 40 +(1 row) + +DROP TABLE join_mcv_semi, join_mcv_semi_indexed; +-- check join statistics on the partitions are used in a partitionwise join +CREATE TABLE join_mcv_part (a int) PARTITION BY RANGE (a); +CREATE TABLE join_mcv_part_1 PARTITION OF join_mcv_part + FOR VALUES FROM (0) TO (100) WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_part_2 PARTITION OF join_mcv_part + FOR VALUES FROM (100) TO (200) WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_part_indexed (a int, y int) PARTITION BY RANGE (a); +CREATE TABLE join_mcv_part_indexed_1 PARTITION OF join_mcv_part_indexed + FOR VALUES FROM (0) TO (100) WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_part_indexed_2 PARTITION OF join_mcv_part_indexed + FOR VALUES FROM (100) TO (200) WITH (autovacuum_enabled = off); +CREATE INDEX ON join_mcv_part_indexed (a); +-- y = 1 in a few rows of each partition +INSERT INTO join_mcv_part_indexed +SELECT g, (g % 100 < 10)::int FROM generate_series(0, 199) g; +-- most rows join the rows where y = 1 +INSERT INTO join_mcv_part +SELECT CASE WHEN g % 10 < 9 THEN g % 10 ELSE g % 100 END + (g % 2) * 100 +FROM generate_series(1, 2000) g; +ANALYZE join_mcv_part, join_mcv_part_indexed; +SET enable_partitionwise_join = on; +-- Without join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_part s + JOIN join_mcv_part_indexed i ON (s.a = i.a) + WHERE i.y = 1'); + estimated | actual +-----------+-------- + 200 | 1820 +(1 row) + +-- With join statistics on the partitions +CREATE STATISTICS join_mcv_part_1_stats (mcv) ON i.y +FROM join_mcv_part_1 s JOIN join_mcv_part_indexed_1 i ON (s.a = i.a); +CREATE STATISTICS join_mcv_part_2_stats (mcv) ON i.y +FROM join_mcv_part_2 s JOIN join_mcv_part_indexed_2 i ON (s.a = i.a); +ANALYZE join_mcv_part_1, join_mcv_part_2; +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_part s + JOIN join_mcv_part_indexed i ON (s.a = i.a) + WHERE i.y = 1'); + estimated | actual +-----------+-------- + 1820 | 1820 +(1 row) + +RESET enable_partitionwise_join; +DROP TABLE join_mcv_part, join_mcv_part_indexed; +-- check ANALYZE of the anchor table builds the statistics, from its sample +-- and the matching rows of the probed table, and ANALYZE of the probed +-- table does not +CREATE STATISTICS join_mcv_movies_probe_stats (mcv) ON k.keyword +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword_id = k.id); +ANALYZE join_mcv_keywords; +SELECT d.stxdmcv IS NOT NULL AS has_mcv +FROM pg_statistic_ext s +LEFT JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +WHERE s.stxname = 'join_mcv_movies_probe_stats'; + has_mcv +--------- + f +(1 row) + +ANALYZE join_mcv_movies; +SELECT d.stxdmcv IS NOT NULL AS has_mcv +FROM pg_statistic_ext s +LEFT JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +WHERE s.stxname = 'join_mcv_movies_probe_stats'; + has_mcv +--------- + t +(1 row) + +DROP STATISTICS join_mcv_movies_probe_stats; +-- check sampling a join value with many matches looks the value up once +-- through the index on the probed table, however many rows have it +CREATE TABLE join_mcv_dups (k int, a int) WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_dups_indexed (k int, b int) + WITH (autovacuum_enabled = off); +CREATE INDEX ON join_mcv_dups_indexed (k); +-- Four rows with join value 1, one with 2, and one NULL +INSERT INTO join_mcv_dups +VALUES (1, 1), (1, 1), (1, 1), (1, 2), (2, 3), (NULL, 4); +-- many matches each for 1 and 2, and rows with 0 that no row joins +INSERT INTO join_mcv_dups_indexed SELECT 1, 3 FROM generate_series(1, 100); +INSERT INTO join_mcv_dups_indexed SELECT 2, 5 FROM generate_series(1, 100); +INSERT INTO join_mcv_dups_indexed SELECT 0, 1 FROM generate_series(1, 100); +ANALYZE join_mcv_dups, join_mcv_dups_indexed; +CREATE STATISTICS join_mcv_dups_stats (mcv) ON s.a, i.b +FROM join_mcv_dups s JOIN join_mcv_dups_indexed i ON (s.k = i.k); +ANALYZE join_mcv_dups; +-- one index scan per join value that is not NULL, each match read once +SELECT pg_stat_force_next_flush(); + pg_stat_force_next_flush +-------------------------- + +(1 row) + +SELECT idx_scan, idx_tup_read, idx_tup_fetch FROM pg_stat_all_indexes + WHERE indexrelid = 'join_mcv_dups_indexed_k_idx'::regclass; + idx_scan | idx_tup_read | idx_tup_fetch +----------+--------------+--------------- + 2 | 200 | 200 +(1 row) + +-- each row with join value 1 pairs with every match of 1, the row with 2 +-- with the matches of 2, and the NULL row with none +SELECT m.values, m.frequency +FROM pg_statistic_ext s +JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +CROSS JOIN LATERAL pg_mcv_list_items(d.stxdmcv) AS m +WHERE s.stxname = 'join_mcv_dups_stats' +ORDER BY m.frequency DESC, m.values; + values | frequency +--------+----------- + {1,3} | 0.6 + {2,3} | 0.2 + {3,5} | 0.2 +(3 rows) + +DROP TABLE join_mcv_dups, join_mcv_dups_indexed; +-- With a hash index on the probed table, a lookup of a join value also +-- returns the matches of every value with the same hash code, and ANALYZE +-- must leave them out. 2775 and 131913 have the same hash code. +SELECT hashint4(2775) = hashint4(131913) AS same_hash_code; + same_hash_code +---------------- + t +(1 row) + +CREATE TABLE join_mcv_hash (k int) WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_hash_indexed (k int) WITH (autovacuum_enabled = off); +CREATE INDEX ON join_mcv_hash_indexed USING hash (k); +INSERT INTO join_mcv_hash VALUES (2775); +INSERT INTO join_mcv_hash_indexed +VALUES (2775), (2775), (2775), (2775), (131913); +ANALYZE join_mcv_hash, join_mcv_hash_indexed; +CREATE STATISTICS join_mcv_hash_stats (mcv) ON i.k +FROM join_mcv_hash s JOIN join_mcv_hash_indexed i ON (s.k = i.k); +ANALYZE join_mcv_hash; +-- only the matches of 2775 +SELECT m.values, m.frequency +FROM pg_statistic_ext s +JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +CROSS JOIN LATERAL pg_mcv_list_items(d.stxdmcv) AS m +WHERE s.stxname = 'join_mcv_hash_stats'; + values | frequency +--------+----------- + {2775} | 1 +(1 row) + +DROP TABLE join_mcv_hash, join_mcv_hash_indexed; +-- check a join larger than the sample: with statistics target 10, ANALYZE +-- samples 1000 of the 100000 joined rows, so the frequencies vary from run +-- to run +CREATE TABLE join_mcv_big (k int) WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_big_indexed (k int, b int) + WITH (autovacuum_enabled = off); +CREATE INDEX ON join_mcv_big_indexed (k); +-- one join value, whose matches have b in 1..4 in equal shares +INSERT INTO join_mcv_big SELECT 1 FROM generate_series(1, 40); +INSERT INTO join_mcv_big_indexed +SELECT 1, (g - 1) / 625 + 1 FROM generate_series(1, 2500) g; +ANALYZE join_mcv_big, join_mcv_big_indexed; +CREATE STATISTICS join_mcv_big_stats (mcv) ON i.b +FROM join_mcv_big s JOIN join_mcv_big_indexed i ON (s.k = i.k); +ALTER STATISTICS join_mcv_big_stats SET STATISTICS 10; +ANALYZE join_mcv_big; +-- the sample has every value of b, not only the first ones, and exactly the +-- rows the target allows, so every frequency is a multiple of 1/1000 +SELECT count(*) AS b_values, + bool_and(abs(m.frequency * 1000 - round(m.frequency * 1000)) < 1e-6) + AS in_thousandths +FROM pg_statistic_ext s +JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +CROSS JOIN LATERAL pg_mcv_list_items(d.stxdmcv) AS m +WHERE s.stxname = 'join_mcv_big_stats'; + b_values | in_thousandths +----------+---------------- + 4 | t +(1 row) + +DROP TABLE join_mcv_big, join_mcv_big_indexed; +-- check the join column of the probed table needs an index ANALYZE can look +-- the join values up in: without one, CREATE warns and ANALYZE skips the +-- object +-- XXX Sampling without an index, by scanning the probed table, is planned; +-- the cases skipped here will then build. +CREATE TABLE join_mcv_noidx (id INTEGER, val TEXT); +-- a few rows, joined by the movies with keyword_id 1 +INSERT INTO join_mcv_noidx SELECT 1, 'x' FROM generate_series(1, 20); +CREATE STATISTICS join_mcv_noidx_stats (mcv) ON n.val +FROM join_mcv_movies m JOIN join_mcv_noidx n ON (m.keyword_id = n.id); +WARNING: no suitable index on "join_mcv_noidx" column "id" for join statistics +DETAIL: ANALYZE skips the statistics object while the table has no suitable index. +HINT: Create an index on the join column to enable index-based join sampling. +ANALYZE join_mcv_movies; +WARNING: skipping join statistics "public.join_mcv_noidx_stats" +DETAIL: Table "public.join_mcv_noidx" has no suitable index on column "id". +SELECT d.stxdmcv IS NOT NULL AS has_mcv +FROM pg_statistic_ext s +LEFT JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +WHERE s.stxname = 'join_mcv_noidx_stats'; + has_mcv +--------- + f +(1 row) + +-- None of these is suitable: a BRIN index does not support equality lookup, +-- a partial index does not cover every row, and a multi-column index can be +-- looked up only by its leading column. +CREATE INDEX join_mcv_noidx_brin ON join_mcv_noidx USING brin (id); +CREATE INDEX join_mcv_noidx_partial ON join_mcv_noidx (id) WHERE id > 0; +CREATE INDEX join_mcv_noidx_nonlead ON join_mcv_noidx (val, id); +ANALYZE join_mcv_movies; +WARNING: skipping join statistics "public.join_mcv_noidx_stats" +DETAIL: Table "public.join_mcv_noidx" has no suitable index on column "id". +SELECT d.stxdmcv IS NOT NULL AS has_mcv +FROM pg_statistic_ext s +LEFT JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +WHERE s.stxname = 'join_mcv_noidx_stats'; + has_mcv +--------- + f +(1 row) + +-- A btree index with the join column leading is suitable. +CREATE INDEX join_mcv_noidx_wide ON join_mcv_noidx (id, val); +ANALYZE join_mcv_movies; +SELECT d.stxdmcv IS NOT NULL AS has_mcv +FROM pg_statistic_ext s +LEFT JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +WHERE s.stxname = 'join_mcv_noidx_stats'; + has_mcv +--------- + t +(1 row) + +DROP INDEX join_mcv_noidx_wide; +-- With the last usable index gone, ANALYZE warns and keeps the data it has +ANALYZE join_mcv_movies; +WARNING: skipping join statistics "public.join_mcv_noidx_stats" +DETAIL: Table "public.join_mcv_noidx" has no suitable index on column "id". +SELECT d.stxdmcv IS NOT NULL AS has_mcv +FROM pg_statistic_ext s +LEFT JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +WHERE s.stxname = 'join_mcv_noidx_stats'; + has_mcv +--------- + t +(1 row) + +DROP TABLE join_mcv_noidx; +-- check ALTER TABLE ... ALTER COLUMN TYPE re-creates the statistics objects +-- that use the column, as for single-table statistics, and leaves the +-- others alone +CREATE STATISTICS join_mcv_movies_alter_stats (mcv) ON mk.year +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword_id = k.id); +ANALYZE join_mcv_movies; +ALTER TABLE join_mcv_keywords ALTER COLUMN phonetic_code TYPE varchar(10); +SELECT d.stxdmcv IS NOT NULL AS has_mcv +FROM pg_statistic_ext s +LEFT JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +WHERE s.stxname = 'join_mcv_movies_alter_stats'; + has_mcv +--------- + t +(1 row) + +-- A column of the statistics: re-created without data +ALTER TABLE join_mcv_movies ALTER COLUMN year TYPE numeric; +SELECT d.stxdmcv IS NOT NULL AS has_mcv +FROM pg_statistic_ext s +LEFT JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +WHERE s.stxname = 'join_mcv_movies_alter_stats'; + has_mcv +--------- + f +(1 row) + +DROP STATISTICS join_mcv_movies_alter_stats; +-- Changing the type of a column of the probed table re-creates the +-- statistics, as for a column of the table they belong to +CREATE STATISTICS join_mcv_movies_alter_dim_stats (mcv) ON k.keyword +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword_id = k.id); +ANALYZE join_mcv_movies; +SELECT pg_get_statisticsobjdef(s.oid, false), d.stxdmcv IS NOT NULL AS has_mcv +FROM pg_statistic_ext s LEFT JOIN pg_statistic_ext_data d ON d.stxoid = s.oid +WHERE s.stxname = 'join_mcv_movies_alter_dim_stats'; + pg_get_statisticsobjdef | has_mcv +----------------------------------------------------------------------------------------------------------------------------------------------------------------+--------- + join_mcv_keywords.keyword FROM join_mcv_movies join_mcv_movies JOIN join_mcv_keywords join_mcv_keywords ON (join_mcv_movies.keyword_id = join_mcv_keywords.id) | t +(1 row) + +BEGIN; +ALTER TABLE join_mcv_keywords ALTER COLUMN keyword TYPE varchar(20); +-- ALTER TABLE does not lock join_mcv_movies exclusively: it is locked only +-- as for CREATE STATISTICS +SELECT mode FROM pg_locks +WHERE pid = pg_backend_pid() AND relation = 'join_mcv_movies'::regclass +ORDER BY mode; + mode +-------------------------- + AccessShareLock + ShareUpdateExclusiveLock +(2 rows) + +COMMIT; +SELECT pg_get_statisticsobjdef(s.oid, false), d.stxdmcv IS NOT NULL AS has_mcv +FROM pg_statistic_ext s LEFT JOIN pg_statistic_ext_data d ON d.stxoid = s.oid +WHERE s.stxname = 'join_mcv_movies_alter_dim_stats'; + pg_get_statisticsobjdef | has_mcv +----------------------------------------------------------------------------------------------------------------------------------------------------------------+--------- + join_mcv_keywords.keyword FROM join_mcv_movies join_mcv_movies JOIN join_mcv_keywords join_mcv_keywords ON (join_mcv_movies.keyword_id = join_mcv_keywords.id) | f +(1 row) + +DROP STATISTICS join_mcv_movies_alter_dim_stats; +-- A self-join's statistics are re-created once, although both sides change +CREATE STATISTICS join_mcv_movies_alter_self_stats (mcv) ON b.keyword +FROM join_mcv_keywords a JOIN join_mcv_keywords b ON (a.id = b.id); +ALTER TABLE join_mcv_keywords ALTER COLUMN id TYPE bigint; +SELECT pg_get_statisticsobjdef(oid, false) FROM pg_statistic_ext +WHERE stxname = 'join_mcv_movies_alter_self_stats'; + pg_get_statisticsobjdef +-------------------------------------------------------------------------------------------------------------------------------------------------------------------- + join_mcv_keywords_1.keyword FROM join_mcv_keywords join_mcv_keywords JOIN join_mcv_keywords join_mcv_keywords_1 ON (join_mcv_keywords.id = join_mcv_keywords_1.id) +(1 row) + +DROP STATISTICS join_mcv_movies_alter_self_stats; +-- A join column may change to a type that can still be compared with the +-- other one (join_mcv_keywords.id is bigint since the case above); otherwise +-- ALTER TABLE fails, as the statistics cannot be re-created +CREATE STATISTICS join_mcv_movies_alter_key_stats (mcv) ON k.keyword +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword_id = k.id); +ALTER TABLE join_mcv_movies ALTER COLUMN keyword_id TYPE bigint; +ALTER TABLE join_mcv_movies ALTER COLUMN keyword_id TYPE numeric; +ERROR: join statistics support only equality conditions between columns of two tables +DETAIL: This condition is not an equality between two columns. +DROP STATISTICS join_mcv_movies_alter_key_stats; +DROP TABLE join_mcv_movies, join_mcv_keywords; +-- check tables with inheritance children are refused like partitioned +-- tables, whichever table has them, and children added later make ANALYZE +-- skip the statistics +CREATE TABLE join_mcv_inh (dim_id int); +CREATE TABLE join_mcv_inh_indexed (id int PRIMARY KEY, label text); +INSERT INTO join_mcv_inh_indexed +SELECT g, 'label ' || g % 3 FROM generate_series(1, 99) g; +INSERT INTO join_mcv_inh SELECT 1 + g % 99 FROM generate_series(1, 1000) g; +CREATE TABLE join_mcv_inh_child () INHERITS (join_mcv_inh); +CREATE STATISTICS join_mcv_inh_anchor_stats (mcv) ON d.label +FROM join_mcv_inh f JOIN join_mcv_inh_indexed d ON (f.dim_id = d.id); +ERROR: join statistics do not support tables with inheritance children +DETAIL: Table "join_mcv_inh" has inheritance children. +DROP TABLE join_mcv_inh_child; +CREATE TABLE join_mcv_inh_indexed_child () INHERITS (join_mcv_inh_indexed); +CREATE STATISTICS join_mcv_inh_probed_stats (mcv) ON d.label +FROM join_mcv_inh f JOIN join_mcv_inh_indexed d ON (f.dim_id = d.id); +ERROR: join statistics do not support tables with inheritance children +DETAIL: Table "join_mcv_inh_indexed" has inheritance children. +DROP TABLE join_mcv_inh_indexed_child; +-- Children added later: the statistics are skipped, rather than built +-- without the children's rows +CREATE STATISTICS join_mcv_inh_later_stats (mcv) ON d.label +FROM join_mcv_inh f JOIN join_mcv_inh_indexed d ON (f.dim_id = d.id); +CREATE TABLE join_mcv_inh_indexed_child () INHERITS (join_mcv_inh_indexed); +INSERT INTO join_mcv_inh_indexed_child +SELECT g, 'child' FROM generate_series(1, 99) g; +ANALYZE join_mcv_inh; +WARNING: skipping join statistics "public.join_mcv_inh_later_stats" +DETAIL: Table "public.join_mcv_inh_indexed" has inheritance children. +SELECT most_common_vals FROM pg_stats_ext +WHERE statistics_name = 'join_mcv_inh_later_stats'; + most_common_vals +------------------ +(0 rows) + +-- Children of the anchor table are reported once, although ANALYZE goes over +-- that table both with and without its children +CREATE TABLE join_mcv_inh_child () INHERITS (join_mcv_inh); +ANALYZE join_mcv_inh; +WARNING: skipping join statistics "public.join_mcv_inh_later_stats" +DETAIL: Table "public.join_mcv_inh" has inheritance children. +DROP TABLE join_mcv_inh, join_mcv_inh_indexed CASCADE; +NOTICE: drop cascades to 2 other objects +DETAIL: drop cascades to table join_mcv_inh_indexed_child +drop cascades to table join_mcv_inh_child +-- Permission tests. CREATE requires ownership of the anchor table and +-- SELECT on the probed table. ANALYZE refreshes the statistics only if the +-- owner still has SELECT on the probed table; otherwise it warns and skips +-- them, leaving the existing data in place. +-- +CREATE ROLE regress_join_mcv_user; +CREATE ROLE regress_join_mcv_user2; +CREATE TABLE join_mcv_perm (id int, val int) WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_perm_indexed (id int PRIMARY KEY, label text) + WITH (autovacuum_enabled = off); +INSERT INTO join_mcv_perm SELECT i, i % 10 FROM generate_series(1, 1000) i; +INSERT INTO join_mcv_perm_indexed SELECT i, 'l' FROM generate_series(1, 1000) i; +ANALYZE join_mcv_perm, join_mcv_perm_indexed; +GRANT CREATE ON SCHEMA public TO regress_join_mcv_user, regress_join_mcv_user2; +GRANT SELECT ON join_mcv_perm, join_mcv_perm_indexed TO regress_join_mcv_user; +-- Not the anchor owner: CREATE fails +SET SESSION AUTHORIZATION regress_join_mcv_user; +CREATE STATISTICS join_mcv_perm_stats (mcv) ON a.val +FROM join_mcv_perm a JOIN join_mcv_perm_indexed i ON (a.id = i.id); +ERROR: must be owner of table join_mcv_perm +RESET SESSION AUTHORIZATION; +ALTER TABLE join_mcv_perm OWNER TO regress_join_mcv_user; +-- Anchor owner, no SELECT on the probed table: CREATE fails +REVOKE SELECT ON join_mcv_perm_indexed FROM regress_join_mcv_user; +SET SESSION AUTHORIZATION regress_join_mcv_user; +CREATE STATISTICS join_mcv_perm_stats (mcv) ON a.val +FROM join_mcv_perm a JOIN join_mcv_perm_indexed i ON (a.id = i.id); +ERROR: permission denied for table join_mcv_perm_indexed +RESET SESSION AUTHORIZATION; +-- Anchor owner, SELECT on the probed table: CREATE succeeds +GRANT SELECT ON join_mcv_perm_indexed TO regress_join_mcv_user; +SET SESSION AUTHORIZATION regress_join_mcv_user; +CREATE STATISTICS join_mcv_perm_stats (mcv) ON a.val +FROM join_mcv_perm a JOIN join_mcv_perm_indexed i ON (a.id = i.id); +RESET SESSION AUTHORIZATION; +-- Owner has SELECT: ANALYZE builds the statistics +ANALYZE join_mcv_perm; +SELECT d.stxdmcv IS NOT NULL AS has_mcv +FROM pg_statistic_ext s +LEFT JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +WHERE s.stxname = 'join_mcv_perm_stats'; + has_mcv +--------- + t +(1 row) + +-- Anchor owner changed: ANALYZE still refreshes (no warning), and ALTER +-- COLUMN TYPE still re-creates the object for its owner, without data until +-- the next ANALYZE +ALTER TABLE join_mcv_perm OWNER TO CURRENT_USER; +ANALYZE join_mcv_perm; +ALTER TABLE join_mcv_perm ALTER COLUMN val TYPE bigint; +SELECT stxname, stxowner::regrole FROM pg_statistic_ext +WHERE stxname = 'join_mcv_perm_stats'; + stxname | stxowner +---------------------+----------------------- + join_mcv_perm_stats | regress_join_mcv_user +(1 row) + +ANALYZE join_mcv_perm; +-- Statistics owner changed to a role without SELECT on the probed table: +-- ANALYZE skips +ALTER STATISTICS join_mcv_perm_stats OWNER TO regress_join_mcv_user2; +ANALYZE join_mcv_perm; +WARNING: skipping join statistics "public.join_mcv_perm_stats" +DETAIL: Its owner lacks SELECT privilege on table "public.join_mcv_perm_indexed". +SELECT d.stxdmcv IS NOT NULL AS has_mcv +FROM pg_statistic_ext s +LEFT JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +WHERE s.stxname = 'join_mcv_perm_stats'; + has_mcv +--------- + t +(1 row) + +-- Owner granted SELECT on the probed table: ANALYZE refreshes (no warning) +GRANT SELECT ON join_mcv_perm_indexed TO regress_join_mcv_user2; +ANALYZE join_mcv_perm; +-- Owner's SELECT on the probed table revoked: ANALYZE skips, while ALTER +-- COLUMN TYPE still re-creates the object (no error) +REVOKE SELECT ON join_mcv_perm_indexed FROM regress_join_mcv_user2; +ANALYZE join_mcv_perm; +WARNING: skipping join statistics "public.join_mcv_perm_stats" +DETAIL: Its owner lacks SELECT privilege on table "public.join_mcv_perm_indexed". +SELECT d.stxdmcv IS NOT NULL AS has_mcv +FROM pg_statistic_ext s +LEFT JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +WHERE s.stxname = 'join_mcv_perm_stats'; + has_mcv +--------- + t +(1 row) + +ALTER TABLE join_mcv_perm ALTER COLUMN val TYPE int; +DROP STATISTICS join_mcv_perm_stats; +DROP TABLE join_mcv_perm, join_mcv_perm_indexed; +REVOKE CREATE ON SCHEMA public + FROM regress_join_mcv_user, regress_join_mcv_user2; +DROP ROLE regress_join_mcv_user, regress_join_mcv_user2; +-- check the owner of a joined table can drop their index, Tomas Vondra's +-- scenario: owner1 owns the anchor and the statistics object, owner2 the +-- other table and its index +CREATE ROLE regress_join_mcv_owner1; +CREATE ROLE regress_join_mcv_owner2; +GRANT USAGE, CREATE ON SCHEMA public TO regress_join_mcv_owner1; +GRANT USAGE, CREATE ON SCHEMA public TO regress_join_mcv_owner2; +SET ROLE regress_join_mcv_owner2; +CREATE TABLE join_mcv_owners_indexed (a int, b int); +INSERT INTO join_mcv_owners_indexed +SELECT g, g % 2 FROM generate_series(1, 100) g; +CREATE INDEX join_mcv_owners_idx ON join_mcv_owners_indexed (a); +RESET ROLE; +GRANT SELECT ON join_mcv_owners_indexed TO regress_join_mcv_owner1; +SET ROLE regress_join_mcv_owner1; +CREATE TABLE join_mcv_owners (a int, b int); +INSERT INTO join_mcv_owners +SELECT g % 100 + 1, g % 3 FROM generate_series(1, 1000) g; +CREATE STATISTICS join_mcv_owners_stats (mcv) ON s.a, s.b, i.a, i.b + FROM join_mcv_owners s JOIN join_mcv_owners_indexed i ON (s.a = i.a); +ANALYZE join_mcv_owners; +-- one dependency row per column, although s.a and i.a are both join and +-- statistics columns +SELECT pg_describe_object(refclassid, refobjid, refobjsubid) AS ref +FROM pg_depend +WHERE objid = (SELECT oid FROM pg_statistic_ext + WHERE stxname = 'join_mcv_owners_stats') + AND refobjsubid <> 0 +ORDER BY 1; + ref +------------------------------------------- + column a of table join_mcv_owners + column a of table join_mcv_owners_indexed + column b of table join_mcv_owners + column b of table join_mcv_owners_indexed +(4 rows) + +SET ROLE regress_join_mcv_owner2; +DROP INDEX join_mcv_owners_idx; +-- The statistics object stays, and is skipped until owner2 has an index +-- again +SET ROLE regress_join_mcv_owner1; +ANALYZE join_mcv_owners; +WARNING: skipping join statistics "public.join_mcv_owners_stats" +DETAIL: Table "public.join_mcv_owners_indexed" has no suitable index on column "a". +SELECT stxname FROM pg_statistic_ext WHERE stxname = 'join_mcv_owners_stats'; + stxname +----------------------- + join_mcv_owners_stats +(1 row) + +SET ROLE regress_join_mcv_owner2; +CREATE INDEX join_mcv_owners_idx ON join_mcv_owners_indexed (a); +SET ROLE regress_join_mcv_owner1; +ANALYZE join_mcv_owners; +RESET ROLE; +SELECT d.stxdmcv IS NOT NULL AS has_mcv +FROM pg_statistic_ext s +LEFT JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +WHERE s.stxname = 'join_mcv_owners_stats'; + has_mcv +--------- + t +(1 row) + +DROP TABLE join_mcv_owners, join_mcv_owners_indexed; +REVOKE USAGE, CREATE ON SCHEMA public + FROM regress_join_mcv_owner1, regress_join_mcv_owner2; +DROP ROLE regress_join_mcv_owner1, regress_join_mcv_owner2; +-- check join statistics do not reveal rows that row-level security hides +CREATE FUNCTION op_leak(int, int) RETURNS bool + AS 'BEGIN RAISE NOTICE ''op_leak => %, %'', $1, $2; RETURN $1 < $2; END' + LANGUAGE plpgsql; +-- The join MCV estimator only considers operators that have a restriction +-- estimator and belong to a btree operator family. +CREATE OPERATOR <<< (procedure = op_leak, leftarg = int, rightarg = int, + restrict = scalarltsel); +CREATE OPERATOR FAMILY leak_ops USING btree; +ALTER OPERATOR FAMILY leak_ops USING btree ADD OPERATOR 1 <<< (int4, int4); +CREATE ROLE regress_join_mcv_rls_owner; +CREATE ROLE regress_join_mcv_rls_anchor_owner; +CREATE SCHEMA join_rls AUTHORIZATION regress_join_mcv_rls_owner; +SET ROLE regress_join_mcv_rls_owner; +CREATE TABLE join_rls.anchor (id int, filler int); +CREATE TABLE join_rls.secret (id int, secret text); +INSERT INTO join_rls.anchor SELECT g, g FROM generate_series(1, 100) g; +INSERT INTO join_rls.secret + SELECT g, CASE WHEN g <= 50 THEN 'secret-1' ELSE 'secret-2' END + FROM generate_series(1, 100) g; +CREATE INDEX ON join_rls.secret (id); +-- No column statistics on filler, so op_leak can only see the join MCV's +-- values +ALTER TABLE join_rls.anchor ALTER COLUMN filler SET STATISTICS 0; +CREATE STATISTICS join_rls.secret_stats (mcv) ON anchor.filler, secret.secret + FROM join_rls.anchor JOIN join_rls.secret ON anchor.id = secret.id; +ANALYZE join_rls.anchor; +-- The owner can read secret, so it sees the statistics. +SELECT statistics_name, most_common_vals IS NOT NULL AS has_mcv +FROM pg_stats_ext WHERE statistics_schemaname = 'join_rls'; + statistics_name | has_mcv +-----------------+--------- + secret_stats | t +(1 row) + +-- Owning only anchor is not enough to see them +RESET ROLE; +ALTER TABLE join_rls.anchor OWNER TO regress_join_mcv_rls_anchor_owner; +SET ROLE regress_join_mcv_rls_anchor_owner; +SELECT statistics_name, most_common_vals IS NOT NULL AS has_mcv +FROM pg_stats_ext WHERE statistics_schemaname = 'join_rls'; + statistics_name | has_mcv +-----------------+--------- +(0 rows) + +RESET ROLE; +ALTER TABLE join_rls.anchor OWNER TO regress_join_mcv_rls_owner; +SET ROLE regress_join_mcv_rls_owner; +-- Now nobody can read any rows of secret, not even its owner. (With no +-- policy at all, the planner would prove the join empty and never estimate +-- it.) +ALTER TABLE join_rls.secret ENABLE ROW LEVEL SECURITY; +ALTER TABLE join_rls.secret FORCE ROW LEVEL SECURITY; +CREATE POLICY nobody ON join_rls.secret USING (id < 0); +SELECT count(*) AS visible_secret_rows FROM join_rls.secret; + visible_secret_rows +--------------------- + 0 +(1 row) + +-- So its values must not show up in pg_stats_ext, +SELECT statistics_name, most_common_vals IS NOT NULL AS has_mcv +FROM pg_stats_ext WHERE statistics_schemaname = 'join_rls'; + statistics_name | has_mcv +-----------------+--------- +(0 rows) + +-- nor be passed to a leaky operator while planning: no op_leak NOTICE. +EXPLAIN (COSTS OFF) +SELECT * FROM join_rls.anchor JOIN join_rls.secret ON anchor.id = secret.id + WHERE anchor.filler <<< 5; + QUERY PLAN +-------------------------------------- + Hash Join + Hash Cond: (anchor.id = secret.id) + -> Seq Scan on anchor + Filter: (filler <<< 5) + -> Hash + -> Seq Scan on secret + Filter: (id < 0) +(7 rows) + +RESET ROLE; +DROP SCHEMA join_rls CASCADE; +NOTICE: drop cascades to 2 other objects +DETAIL: drop cascades to table join_rls.anchor +drop cascades to table join_rls.secret +DROP ROLE regress_join_mcv_rls_owner, regress_join_mcv_rls_anchor_owner; +DROP OPERATOR FAMILY leak_ops USING btree; +DROP OPERATOR <<< (int4, int4); +DROP FUNCTION op_leak(int, int); +-- These objects are not dropped, so that the pg_upgrade test dumps and +-- restores join statistics of each of these shapes; the same holds for +-- join_mcv_stored and join_mcv_filters above. +-- +-- check a cross-type join, a float4 anchor column joined to a float8 column +-- through the cross-type equality operator +CREATE TABLE join_mcv_crosstype_indexed (id float8 PRIMARY KEY, + color text NOT NULL) + WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_crosstype (id serial, dim_id float4 NOT NULL) + WITH (autovacuum_enabled = off); +-- three colors +INSERT INTO join_mcv_crosstype_indexed +SELECT i::float8, CASE WHEN i <= 5 THEN 'red' + WHEN i <= 10 THEN 'blue' + ELSE 'green' END +FROM generate_series(1, 15) i; +-- most rows reference the red rows +INSERT INTO join_mcv_crosstype (dim_id) +SELECT CASE WHEN i % 5 < 4 THEN (i % 5 + 1)::float4 + ELSE (6 + i % 10)::float4 END +FROM generate_series(1, 500) i; +ANALYZE join_mcv_crosstype_indexed; +ANALYZE join_mcv_crosstype; +-- Without join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_crosstype f, join_mcv_crosstype_indexed d + WHERE f.dim_id = d.id AND d.color = ''red'' +'); + estimated | actual +-----------+-------- + 167 | 400 +(1 row) + +CREATE STATISTICS join_mcv_crosstype_stats (mcv) ON f.dim_id, d.color +FROM join_mcv_crosstype f + JOIN join_mcv_crosstype_indexed d ON (f.dim_id = d.id); +ANALYZE join_mcv_crosstype; +-- With join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_crosstype f, join_mcv_crosstype_indexed d + WHERE f.dim_id = d.id AND d.color = ''red'' +'); + estimated | actual +-----------+-------- + 400 | 400 +(1 row) + +-- The join condition may name the joined table first: the statistics are the +-- same +CREATE STATISTICS join_mcv_crosstype_swapped_stats (mcv) ON f.dim_id, d.color +FROM join_mcv_crosstype f + JOIN join_mcv_crosstype_indexed d ON (d.id = f.dim_id); +ANALYZE join_mcv_crosstype; +SELECT a.most_common_vals = b.most_common_vals AS same_vals, + a.most_common_freqs = b.most_common_freqs AS same_freqs +FROM pg_stats_ext a, pg_stats_ext b +WHERE a.statistics_name = 'join_mcv_crosstype_stats' + AND b.statistics_name = 'join_mcv_crosstype_swapped_stats'; + same_vals | same_freqs +-----------+------------ + t | t +(1 row) + +DROP STATISTICS join_mcv_crosstype_swapped_stats; +-- Leave join_mcv_crosstype, join_mcv_crosstype_indexed and +-- join_mcv_crosstype_stats in place for pg_upgrade testing. +-- check a join of binary-compatible types, a varchar column joined to a +-- text key: the parser relabels the varchar column to text, as the planner +-- does for the same join in a query +CREATE TABLE join_mcv_relabel_indexed (code text PRIMARY KEY, + color text NOT NULL) + WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_relabel (id serial, dim_code varchar(10) NOT NULL) + WITH (autovacuum_enabled = off); +-- three colors +INSERT INTO join_mcv_relabel_indexed +SELECT 'c' || i, CASE WHEN i <= 5 THEN 'red' + WHEN i <= 10 THEN 'blue' + ELSE 'green' END +FROM generate_series(1, 15) i; +-- most rows reference the red rows +INSERT INTO join_mcv_relabel (dim_code) +SELECT 'c' || CASE WHEN i % 5 < 4 THEN i % 5 + 1 ELSE 6 + i % 10 END +FROM generate_series(1, 500) i; +ANALYZE join_mcv_relabel_indexed; +ANALYZE join_mcv_relabel; +-- Without join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_relabel f, join_mcv_relabel_indexed d + WHERE f.dim_code = d.code AND d.color = ''red'' +'); + estimated | actual +-----------+-------- + 167 | 400 +(1 row) + +CREATE STATISTICS join_mcv_relabel_stats (mcv) ON d.color +FROM join_mcv_relabel f + JOIN join_mcv_relabel_indexed d ON (f.dim_code = d.code); +SELECT pg_get_statisticsobjdef(oid, false) FROM pg_statistic_ext +WHERE stxname = 'join_mcv_relabel_stats'; + pg_get_statisticsobjdef +--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + join_mcv_relabel_indexed.color FROM join_mcv_relabel join_mcv_relabel JOIN join_mcv_relabel_indexed join_mcv_relabel_indexed ON (join_mcv_relabel.dim_code = join_mcv_relabel_indexed.code) +(1 row) + +-- Dependencies are recorded on both join columns, the varchar one included +SELECT pg_describe_object(refclassid, refobjid, refobjsubid) AS ref +FROM pg_depend +WHERE objid = (SELECT oid FROM pg_statistic_ext + WHERE stxname = 'join_mcv_relabel_stats') + AND refobjsubid <> 0 +ORDER BY 1; + ref +------------------------------------------------ + column code of table join_mcv_relabel_indexed + column color of table join_mcv_relabel_indexed + column dim_code of table join_mcv_relabel +(3 rows) + +-- The join condition is stored with its collation +SELECT substring(stxjoinconds from ':inputcollid \d+') AS inputcollid +FROM pg_statistic_ext WHERE stxname = 'join_mcv_relabel_stats'; + inputcollid +------------------ + :inputcollid 100 +(1 row) + +-- A column merged by USING can be used by a later join: it stands for the +-- varchar column +CREATE TABLE join_mcv_relabel_code (code varchar(10)); +CREATE STATISTICS join_mcv_relabel_using_stats (mcv) ON d2.color +FROM join_mcv_relabel_code JOIN join_mcv_relabel_indexed USING (code) + JOIN join_mcv_relabel_indexed d2 USING (code); +SELECT pg_get_statisticsobjdef(oid, false) FROM pg_statistic_ext +WHERE stxname = 'join_mcv_relabel_using_stats'; + pg_get_statisticsobjdef +------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + join_mcv_relabel_indexed_1.color FROM join_mcv_relabel_code join_mcv_relabel_code JOIN join_mcv_relabel_indexed join_mcv_relabel_indexed ON (join_mcv_relabel_code.code = join_mcv_relabel_indexed.code) JOIN join_mcv_relabel_indexed join_mcv_relabel_indexed_1 ON (join_mcv_relabel_code.code = join_mcv_relabel_indexed_1.code) +(1 row) + +DROP TABLE join_mcv_relabel_code; +ANALYZE join_mcv_relabel; +-- With join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_relabel f, join_mcv_relabel_indexed d + WHERE f.dim_code = d.code AND d.color = ''red'' +'); + estimated | actual +-----------+-------- + 400 | 400 +(1 row) + +-- LIKE INCLUDING STATISTICS copies only the single-table statistics: join +-- statistics describe a join, not the table +CREATE STATISTICS join_mcv_relabel_single_stats ON id, dim_code +FROM join_mcv_relabel; +CREATE TABLE join_mcv_relabel_copy + (LIKE join_mcv_relabel INCLUDING STATISTICS); +SELECT stxname, pg_get_statisticsobjdef(oid, false) FROM pg_statistic_ext +WHERE stxrelid = 'join_mcv_relabel_copy'::regclass; + stxname | pg_get_statisticsobjdef +----------------------------------------+----------------------------------------- + join_mcv_relabel_copy_id_dim_code_stat | id, dim_code FROM join_mcv_relabel_copy +(1 row) + +DROP TABLE join_mcv_relabel_copy; +DROP STATISTICS join_mcv_relabel_single_stats; +-- Leave join_mcv_relabel, join_mcv_relabel_indexed and join_mcv_relabel_stats +-- in place for pg_upgrade testing. +-- check same-named tables in different schemas get distinct aliases in the +-- deparsed definition +CREATE SCHEMA join_mcv_schema1; +CREATE SCHEMA join_mcv_schema2; +CREATE TABLE join_mcv_schema1.t (a int PRIMARY KEY) + WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_schema2.t (b int) + WITH (autovacuum_enabled = off); +CREATE INDEX ON join_mcv_schema2.t (b); +INSERT INTO join_mcv_schema1.t SELECT generate_series(1, 100); +INSERT INTO join_mcv_schema2.t SELECT generate_series(1, 100); +ANALYZE join_mcv_schema1.t; +ANALYZE join_mcv_schema2.t; +CREATE STATISTICS join_mcv_schemas_stats (mcv) ON join_mcv_schema2.t.b +FROM join_mcv_schema1.t + JOIN join_mcv_schema2.t ON (join_mcv_schema1.t.a = join_mcv_schema2.t.b); +SELECT pg_get_statisticsobjdef(oid) FROM pg_statistic_ext +WHERE stxname = 'join_mcv_schemas_stats'; + pg_get_statisticsobjdef +--------------------------------------------------------------------------------------------------------------------------------------- + CREATE STATISTICS public.join_mcv_schemas_stats (mcv) ON t_1.b FROM join_mcv_schema1.t t JOIN join_mcv_schema2.t t_1 ON (t.a = t_1.b) +(1 row) + +-- Leave the join_mcv_schema1 and join_mcv_schema2 schemas and +-- join_mcv_schemas_stats in place for pg_upgrade testing. +-- check the auto-generated name of a join statistics object qualifies +-- columns with their relation, written before the first column and again +-- whenever the relation changes (single-table naming is covered in +-- stats_ext) +CREATE TABLE join_mcv_auto1 (a INTEGER, b INTEGER, c INTEGER); +CREATE TABLE join_mcv_auto2 (d INTEGER, e INTEGER, f INTEGER); +CREATE INDEX ON join_mcv_auto1 (a); +CREATE INDEX ON join_mcv_auto2 (d); +-- Two columns from each relation: each relation name is written once. +CREATE STATISTICS (mcv) ON a1.b, a1.c, a2.e, a2.f + FROM join_mcv_auto1 a1 JOIN join_mcv_auto2 a2 ON (a1.a = a2.d); +-- Columns alternating between the relations: the relation name is written +-- again each time it changes. +CREATE STATISTICS (mcv) ON a2.e, a1.b, a2.f + FROM join_mcv_auto1 a1 JOIN join_mcv_auto2 a2 ON (a1.a = a2.d); +-- Columns from only one of the relations: that relation name is written once. +CREATE STATISTICS (mcv) ON a2.e, a2.f + FROM join_mcv_auto1 a1 JOIN join_mcv_auto2 a2 ON (a1.a = a2.d); +\dX join_mcv_auto* + List of extended statistics + Schema | Name | Definition | Ndistinct | Dependencies | MCV +--------+---------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+--------------+--------- + public | join_mcv_auto1_b_c_join_mcv_auto2_e_f_stat | join_mcv_auto1.b, join_mcv_auto1.c, join_mcv_auto2.e, join_mcv_auto2.f FROM join_mcv_auto1 join_mcv_auto1 JOIN join_mcv_auto2 join_mcv_auto2 ON (join_mcv_auto1.a = join_mcv_auto2.d) | | | defined + public | join_mcv_auto2_e_f_stat | join_mcv_auto2.e, join_mcv_auto2.f FROM join_mcv_auto1 join_mcv_auto1 JOIN join_mcv_auto2 join_mcv_auto2 ON (join_mcv_auto1.a = join_mcv_auto2.d) | | | defined + public | join_mcv_auto2_e_join_mcv_auto1_b_join_mcv_auto2_f_stat | join_mcv_auto2.e, join_mcv_auto1.b, join_mcv_auto2.f FROM join_mcv_auto1 join_mcv_auto1 JOIN join_mcv_auto2 join_mcv_auto2 ON (join_mcv_auto1.a = join_mcv_auto2.d) | | | defined +(3 rows) + +-- Leave join_mcv_auto1, join_mcv_auto2 and their statistics in place for +-- pg_upgrade testing. +-- check definitions printed differently than written: tables in another +-- order, and several join conditions +CREATE TABLE join_mcv_shape (id INTEGER PRIMARY KEY, dim1_id INTEGER); +CREATE TABLE join_mcv_shape_dim1 (id INTEGER PRIMARY KEY, val TEXT); +CREATE TABLE join_mcv_shape_dim2 (id INTEGER PRIMARY KEY, label TEXT); +CREATE STATISTICS join_mcv_shape_nested_stats (mcv) ON d1.val, d2.label + FROM join_mcv_shape f + JOIN (join_mcv_shape_dim1 d1 JOIN join_mcv_shape_dim2 d2 ON (d1.id = d2.id)) + ON (f.dim1_id = d2.id); +CREATE STATISTICS join_mcv_shape_multiclause_stats (mcv) ON d1.val + FROM join_mcv_shape f + JOIN join_mcv_shape_dim1 d1 ON (f.dim1_id = d1.id AND f.id = d1.id); +SELECT stxname, stxjoinrels::regclass[], pg_get_statisticsobjdef(oid, false) +FROM pg_statistic_ext WHERE stxname LIKE 'join_mcv_shape%' ORDER BY stxname; + stxname | stxjoinrels | pg_get_statisticsobjdef +----------------------------------+----------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + join_mcv_shape_multiclause_stats | [0:1]={join_mcv_shape,join_mcv_shape_dim1} | join_mcv_shape_dim1.val FROM join_mcv_shape join_mcv_shape JOIN join_mcv_shape_dim1 join_mcv_shape_dim1 ON (join_mcv_shape.dim1_id = join_mcv_shape_dim1.id AND join_mcv_shape.id = join_mcv_shape_dim1.id) + join_mcv_shape_nested_stats | [0:2]={join_mcv_shape,join_mcv_shape_dim2,join_mcv_shape_dim1} | join_mcv_shape_dim1.val, join_mcv_shape_dim2.label FROM join_mcv_shape join_mcv_shape JOIN join_mcv_shape_dim2 join_mcv_shape_dim2 ON (join_mcv_shape.dim1_id = join_mcv_shape_dim2.id) JOIN join_mcv_shape_dim1 join_mcv_shape_dim1 ON (join_mcv_shape_dim1.id = join_mcv_shape_dim2.id) +(2 rows) + +-- Leave join_mcv_shape, join_mcv_shape_dim1, join_mcv_shape_dim2 and their +-- statistics in place for pg_upgrade testing. diff --git a/src/test/regress/expected/stats_import.out b/src/test/regress/expected/stats_import.out index 0702ae3e464..1dfa0cfa000 100644 --- a/src/test/regress/expected/stats_import.out +++ b/src/test/regress/expected/stats_import.out @@ -4636,8 +4636,154 @@ ORDER BY expr; (a + b) | 3 (2 rows) +-- Join statistics: copy their statistics to clones and compare, like for +-- test_stat above. The columns of join statistics can come from the joined +-- table rather than from the table the statistics belong to. +CREATE TABLE stats_import.tail_dim ( + id INTEGER PRIMARY KEY, + label TEXT NOT NULL +) WITH (autovacuum_enabled = false); +CREATE TABLE stats_import.tail_fact ( + dim_id INTEGER NOT NULL +) WITH (autovacuum_enabled = false); +INSERT INTO stats_import.tail_dim VALUES + (1, 'common_a'), (2, 'common_b'), + (3, 'other_a'), (4, 'other_b'), + (5, 'extra_1'), (6, 'extra_2'); +INSERT INTO stats_import.tail_fact (dim_id) +SELECT 1 FROM generate_series(1, 90) +UNION ALL SELECT 2 FROM generate_series(1, 70) +UNION ALL SELECT 3 FROM generate_series(1, 25) +UNION ALL SELECT 4 FROM generate_series(1, 15); +ANALYZE stats_import.tail_dim; +ANALYZE stats_import.tail_fact; +-- tail_dim.label is column 2, and tail_fact has only one column +CREATE STATISTICS stats_import.jstats_tail (mcv) ON d.label +FROM stats_import.tail_fact f JOIN stats_import.tail_dim d ON (f.dim_id = d.id); +ALTER STATISTICS stats_import.jstats_tail SET STATISTICS 2; +ANALYZE stats_import.tail_fact; +CREATE TABLE stats_import.test_orders (id int, test_id int, qty int) + WITH (autovacuum_enabled = false); +INSERT INTO stats_import.test_orders + SELECT g, g % 4 + 1, g % 2 FROM generate_series(1, 80) g; +-- Columns from both tables; test.name is column 2, which in test_orders is +-- the integer test_id +CREATE STATISTICS stats_import.test_orders_stat (mcv) ON o.qty, t.name +FROM stats_import.test_orders o JOIN stats_import.test t ON (o.test_id = t.id); +ANALYZE stats_import.test_orders; +CREATE TABLE stats_import.tail_fact_clone (LIKE stats_import.tail_fact) + WITH (autovacuum_enabled = false); +CREATE TABLE stats_import.tail_dim_clone (LIKE stats_import.tail_dim) + WITH (autovacuum_enabled = false); +CREATE TABLE stats_import.test_orders_clone (LIKE stats_import.test_orders) + WITH (autovacuum_enabled = false); +-- Join statistics need an index on the joined table's join column +CREATE INDEX ON stats_import.tail_dim_clone (id); +CREATE INDEX ON stats_import.test_clone (id); +CREATE STATISTICS stats_import.jstats_tail_clone (mcv) ON d.label +FROM stats_import.tail_fact_clone f JOIN stats_import.tail_dim_clone d + ON (f.dim_id = d.id); +CREATE STATISTICS stats_import.test_orders_stat_clone (mcv) ON o.qty, t.name +FROM stats_import.test_orders_clone o JOIN stats_import.test_clone t + ON (o.test_id = t.id); +-- The statistics to copy +SELECT statistics_name, most_common_vals, most_common_freqs FROM pg_stats_ext +WHERE statistics_schemaname = 'stats_import' + AND statistics_name IN ('jstats_tail', 'test_orders_stat') +ORDER BY statistics_name; + statistics_name | most_common_vals | most_common_freqs +------------------+------------------------------------+----------------------- + jstats_tail | {{common_a},{common_b}} | {0.45,0.35} + test_orders_stat | {{0,one},{0,tre},{1,four},{1,two}} | {0.25,0.25,0.25,0.25} +(2 rows) + +SELECT pg_catalog.pg_restore_extended_stats( + 'schemaname', 'stats_import', + 'relname', 'tail_fact_clone', + 'statistics_schemaname', 'stats_import', + 'statistics_name', 'jstats_tail_clone', + 'inherited', e.inherited, + 'most_common_vals', e.most_common_vals, + 'most_common_freqs', e.most_common_freqs, + 'most_common_base_freqs', e.most_common_base_freqs) +FROM pg_stats_ext AS e +WHERE e.statistics_schemaname = 'stats_import' + AND e.statistics_name = 'jstats_tail'; + pg_restore_extended_stats +--------------------------- + t +(1 row) + +SELECT pg_catalog.pg_restore_extended_stats( + 'schemaname', 'stats_import', + 'relname', 'test_orders_clone', + 'statistics_schemaname', 'stats_import', + 'statistics_name', 'test_orders_stat_clone', + 'inherited', e.inherited, + 'most_common_vals', e.most_common_vals, + 'most_common_freqs', e.most_common_freqs, + 'most_common_base_freqs', e.most_common_base_freqs) +FROM pg_stats_ext AS e +WHERE e.statistics_schemaname = 'stats_import' + AND e.statistics_name = 'test_orders_stat'; + pg_restore_extended_stats +--------------------------- + t +(1 row) + +-- The clones have the same statistics: no differences +SELECT statname, (stats).* +FROM stats_import.pg_stats_ext_get_difference('jstats_tail', 'jstats_tail_clone') +\gx +(0 rows) + +SELECT statname, (stats).* +FROM stats_import.pg_stats_ext_get_difference('test_orders_stat', + 'test_orders_stat_clone') +\gx +(0 rows) + +-- Restoring join statistics needs MAINTAIN on the anchor table only: it +-- writes the given values and reads nothing from the other tables +CREATE ROLE regress_test_join_restore; +GRANT USAGE ON SCHEMA stats_import TO regress_test_join_restore; +GRANT MAINTAIN ON stats_import.tail_fact_clone TO regress_test_join_restore; +SET ROLE regress_test_join_restore; +SELECT has_table_privilege('stats_import.tail_dim_clone', 'SELECT, MAINTAIN') + AS dim_privileges; + dim_privileges +---------------- + f +(1 row) + +SELECT pg_catalog.pg_restore_extended_stats( + 'schemaname', 'stats_import', + 'relname', 'tail_fact_clone', + 'statistics_schemaname', 'stats_import', + 'statistics_name', 'jstats_tail_clone', + 'inherited', false, + 'most_common_vals', '{{common_a}}'::text[], + 'most_common_freqs', '{1}'::double precision[], + 'most_common_base_freqs', '{1}'::double precision[]); + pg_restore_extended_stats +--------------------------- + t +(1 row) + +RESET ROLE; +SELECT most_common_vals FROM pg_stats_ext +WHERE statistics_schemaname = 'stats_import' + AND statistics_name = 'jstats_tail_clone'; + most_common_vals +------------------ + {{common_a}} +(1 row) + +REVOKE MAINTAIN ON stats_import.tail_fact_clone FROM regress_test_join_restore; +REVOKE USAGE ON SCHEMA stats_import FROM regress_test_join_restore; +DROP ROLE regress_test_join_restore; DROP SCHEMA stats_import CASCADE; -NOTICE: drop cascades to 42 other objects +NOTICE: drop cascades to 48 other objects DETAIL: drop cascades to view stats_import.pg_stats_stable drop cascades to view stats_import.pg_statistic_flat_t drop cascades to function stats_import.pg_statistic_flat(text) @@ -4680,3 +4826,9 @@ drop cascades to table stats_import.test_reentrant_attr2 drop cascades to function stats_import.drop_during_import_attr(uuid) drop cascades to type stats_import.reentrant_dom_attr2 drop cascades to table stats_import.test_fold +drop cascades to table stats_import.tail_dim +drop cascades to table stats_import.tail_fact +drop cascades to table stats_import.test_orders +drop cascades to table stats_import.tail_fact_clone +drop cascades to table stats_import.tail_dim_clone +drop cascades to table stats_import.test_orders_clone diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index bb13a7b6b8f..f83226d0b31 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -83,6 +83,9 @@ test: create_table_like alter_generic alter_operator misc async dbsize merge mis # amutils depends on geometry, create_index_spgist, hash_index, brin test: rules psql psql_crosstab psql_pipeline amutils stats_ext collate.linux.utf8 collate.windows.win1252 +# join MCV statistics tests +test: stats_ext_crossrel + # ---------- # Run these alone so they don't run out of parallel workers # select_parallel depends on create_misc diff --git a/src/test/regress/sql/stats_ext.sql b/src/test/regress/sql/stats_ext.sql index 02116ed3b7a..e0d350d612e 100644 --- a/src/test/regress/sql/stats_ext.sql +++ b/src/test/regress/sql/stats_ext.sql @@ -2056,13 +2056,16 @@ DROP TABLE declared_order_grouping; -- Auto-generated statistics object names. -- -- A simple column reference contributes the column name, whether written --- unqualified or wrapped in parens; a more complex expression contributes a --- fixed "expr". +-- unqualified, table-qualified, or wrapped in parens; a more complex expression +-- contributes a fixed "expr". CREATE TABLE auto_t1 (a INTEGER, b INTEGER, c INTEGER); -- Unqualified columns: bare column names. CREATE STATISTICS ON b, c FROM auto_t1; +-- Table-qualified columns: still the column names. +CREATE STATISTICS ON auto_t1.a, auto_t1.b FROM auto_t1; + -- Table-qualified columns in parens: still the column names. CREATE STATISTICS ON (auto_t1.a), (auto_t1.b) FROM auto_t1; diff --git a/src/test/regress/sql/stats_ext_crossrel.sql b/src/test/regress/sql/stats_ext_crossrel.sql new file mode 100644 index 00000000000..964d28d9de1 --- /dev/null +++ b/src/test/regress/sql/stats_ext_crossrel.sql @@ -0,0 +1,2109 @@ +-- Join MCV statistics +-- +-- Note: tables for which we check estimated row counts should be created +-- with autovacuum_enabled = off, so that we don't have unstable results +-- from auto-analyze happening when we didn't expect it. +-- + +-- Each movie has one keyword out of 50: a few keywords are on many movies +-- and most on few or none, which the per-column statistics do not see. +-- Each keyword has its own phonetic_code, NULL for some, and the keyword +-- decides the year. +CREATE TABLE join_mcv_keywords ( + id INTEGER PRIMARY KEY, + keyword TEXT NOT NULL, + phonetic_code character varying(5) +) WITH (autovacuum_enabled = off); + +-- keyword_id has no foreign key to join_mcv_keywords: the join is estimated +-- from the columns' statistics +CREATE TABLE join_mcv_movies ( + movie_id INTEGER PRIMARY KEY, + keyword_id INTEGER NOT NULL, + year INTEGER NOT NULL +) WITH (autovacuum_enabled = off); + +INSERT INTO join_mcv_keywords (id, keyword, phonetic_code) +SELECT + i, + 'keyword_' || i, + CASE WHEN i BETWEEN 26 AND 30 THEN NULL ELSE 'ph_' || i END +FROM generate_series(1, 50) i; + +INSERT INTO join_mcv_movies (movie_id, keyword_id, year) +SELECT + i, + CASE + WHEN i % 100 < 60 THEN (i % 10) + 1 + WHEN i % 100 < 90 THEN (i % 10) + 11 + ELSE (i % 10) + 21 + END, + CASE + WHEN i % 100 < 60 THEN 2020 + WHEN i % 100 < 90 THEN 2021 + ELSE 2022 + END +FROM generate_series(1, 10000) i; +ANALYZE join_mcv_keywords, join_mcv_movies; + +-- The first table of the FROM clause is the anchor: the statistics object +-- belongs to it, and ANALYZE on it samples the join. The definition as +-- stored: the anchor in stxrelid, every table in stxjoinrels, and the mcv +-- kind +CREATE STATISTICS join_mcv_movies_keyword_stats (mcv) +ON k.keyword +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword_id = k.id); +SELECT stxrelid::regclass, stxjoinrels::regclass[], stxkind +FROM pg_statistic_ext WHERE stxname = 'join_mcv_movies_keyword_stats'; +SELECT pg_get_statisticsobjdef(oid, false) FROM pg_statistic_ext +WHERE stxname = 'join_mcv_movies_keyword_stats'; +\dX+ join_mcv_movies_keyword_stats +-- the anchor table lists its join statistics +\d+ join_mcv_movies +-- pg_stats_ext resolves the column in join_mcv_keywords, not by attribute +-- number in the anchor, where 2 is keyword_id +ANALYZE join_mcv_movies; +SELECT exprs FROM pg_stats_ext +WHERE statistics_name = 'join_mcv_movies_keyword_stats'; +DROP STATISTICS join_mcv_movies_keyword_stats; + +-- without join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_movies mk, join_mcv_keywords k + WHERE k.keyword = ''keyword_1'' AND k.id = mk.keyword_id +'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_movies mk, join_mcv_keywords k + WHERE k.keyword IN (''keyword_1'', ''keyword_2'', ''keyword_3'') + AND k.id = mk.keyword_id +'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_movies mk, join_mcv_keywords k + WHERE k.keyword = ''keyword_1'' + AND k.phonetic_code = ''ph_15'' + AND k.id = mk.keyword_id +'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_movies mk, join_mcv_keywords k + WHERE k.phonetic_code = ''ph_1'' AND k.id = mk.keyword_id +'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_movies mk, join_mcv_keywords k + WHERE k.keyword = ''keyword_1'' AND mk.year = 2020 + AND k.id = mk.keyword_id +'); + +-- join statistics on the keyword: each joined keyword with its fraction of +-- the 10000 joined rows +CREATE STATISTICS join_mcv_movies_keyword_stats (mcv) +ON k.keyword +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword_id = k.id); +ANALYZE join_mcv_movies; +SELECT m.values, + m.nulls, + ROUND(m.frequency::numeric, 2) AS frequency +FROM pg_statistic_ext s +JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +CROSS JOIN LATERAL pg_mcv_list_items(d.stxdmcv) AS m +WHERE s.stxname = 'join_mcv_movies_keyword_stats' +ORDER BY m.frequency DESC, m.values; +-- A stored keyword is estimated from its frequency, alone and in a list +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_movies mk, join_mcv_keywords k + WHERE k.keyword = ''keyword_1'' AND k.id = mk.keyword_id +'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_movies mk, join_mcv_keywords k + WHERE k.keyword IN (''keyword_1'', ''keyword_2'', ''keyword_3'') + AND k.id = mk.keyword_id +'); +-- A filter the statistics do not cover is estimated as without them +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_movies mk, join_mcv_keywords k + WHERE k.phonetic_code = ''ph_1'' AND k.id = mk.keyword_id +'); + +-- Join statistics on keyword and phonetic_code store every combination in +-- the join +CREATE STATISTICS join_mcv_movies_keyword_code_stats (mcv) +ON k.keyword, k.phonetic_code +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword_id = k.id); +ANALYZE join_mcv_movies; +SELECT m.values, + m.nulls, + ROUND(m.frequency::numeric, 2) AS frequency +FROM pg_statistic_ext s +JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +CROSS JOIN LATERAL pg_mcv_list_items(d.stxdmcv) AS m +WHERE s.stxname = 'join_mcv_movies_keyword_code_stats' +ORDER BY m.frequency DESC, m.values; +-- phonetic_code is covered now +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_movies mk, join_mcv_keywords k + WHERE k.phonetic_code = ''ph_1'' AND k.id = mk.keyword_id +'); +-- Both columns +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_movies mk, join_mcv_keywords k + WHERE k.keyword = ''keyword_1'' + AND k.phonetic_code = ''ph_1'' + AND k.id = mk.keyword_id +'); +-- keyword_1 never pairs with ph_15, and the statistics store every +-- combination in the join, so the estimate is the one-row minimum +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_movies mk, join_mcv_keywords k + WHERE k.keyword = ''keyword_1'' + AND k.phonetic_code = ''ph_15'' + AND k.id = mk.keyword_id +'); + +-- Still no rows with a filter the statistics do not cover added +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_movies mk, join_mcv_keywords k + WHERE k.keyword = ''keyword_1'' + AND k.phonetic_code = ''ph_15'' + AND mk.year = 2020 + AND k.id = mk.keyword_id +'); + +-- Join statistics with a column from each table +CREATE STATISTICS join_mcv_movies_keyword_year_stats (mcv) +ON k.keyword, mk.year +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword_id = k.id); +ANALYZE join_mcv_movies; +SELECT pg_get_statisticsobjdef(oid, false) FROM pg_statistic_ext +WHERE stxname = 'join_mcv_movies_keyword_year_stats'; +SELECT m.values, + m.nulls, + ROUND(m.frequency::numeric, 2) AS frequency +FROM pg_statistic_ext s +JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +CROSS JOIN LATERAL pg_mcv_list_items(d.stxdmcv) AS m +WHERE s.stxname = 'join_mcv_movies_keyword_year_stats' +ORDER BY m.frequency DESC, m.values; +-- keyword_1 always goes with 2020, which the statistics store +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_movies mk, join_mcv_keywords k + WHERE k.keyword = ''keyword_1'' AND mk.year = 2020 + AND k.id = mk.keyword_id +'); + +-- keyword_1 never goes with 2021 +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_movies mk, join_mcv_keywords k + WHERE k.keyword = ''keyword_1'' AND mk.year = 2021 + AND k.id = mk.keyword_id +'); +-- The columns may be listed in either order +CREATE STATISTICS join_mcv_movies_year_keyword_stats (mcv) +ON mk.year, k.keyword +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword_id = k.id); +ANALYZE join_mcv_movies; +SELECT s.stxname, pg_get_statisticsobjdef(s.oid, false) +FROM pg_statistic_ext s +WHERE s.stxname = 'join_mcv_movies_year_keyword_stats'; +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_movies mk, join_mcv_keywords k + WHERE k.keyword = ''keyword_1'' AND mk.year = 2020 + AND k.id = mk.keyword_id +'); +DROP STATISTICS join_mcv_movies_year_keyword_stats; +DROP STATISTICS join_mcv_movies_keyword_code_stats, + join_mcv_movies_keyword_year_stats; + +-- Verify failures +CREATE STATISTICS join_mcv_bad_no_from (mcv) ON k.keyword; +-- a FROM list with a comma is not supported yet +CREATE STATISTICS join_mcv_bad_comma_list (mcv) ON k.keyword +FROM join_mcv_movies mk, join_mcv_keywords k; +-- nor one with a join among its items +CREATE STATISTICS join_mcv_bad_comma (mcv) ON k.keyword +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword_id = k.id), + join_mcv_movies mk2; +-- no columns +CREATE STATISTICS join_mcv_bad_no_columns (mcv) +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword_id = k.id); +-- an unqualified column name +CREATE STATISTICS join_mcv_bad_unqualified (mcv) ON keyword +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword_id = k.id); +-- an expression +CREATE STATISTICS join_mcv_bad_expression (mcv) ON lower(k.keyword) +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword_id = k.id); +-- other kinds than mcv +CREATE STATISTICS join_mcv_bad_ndistinct (ndistinct) ON k.keyword, mk.year +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword_id = k.id); +-- a condition on a single table would make the statistics describe only the +-- rows satisfying it, alone or with a join condition +CREATE STATISTICS join_mcv_bad_same_table (mcv) ON k.keyword +FROM join_mcv_movies mk + JOIN join_mcv_keywords k ON (mk.keyword_id = mk.movie_id); +CREATE STATISTICS join_mcv_bad_single_table_filter (mcv) ON k.keyword +FROM join_mcv_movies mk + JOIN join_mcv_keywords k ON (mk.keyword_id = k.id + AND mk.movie_id = mk.keyword_id); +-- a non-equality join condition +CREATE STATISTICS join_mcv_bad_non_eq (mcv) ON k.keyword +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword_id > k.id); +-- an outer join +CREATE STATISTICS join_mcv_bad_left (mcv) ON k.keyword +FROM join_mcv_movies mk LEFT JOIN join_mcv_keywords k ON (mk.keyword_id = k.id); +-- a whole-row reference in the join condition +CREATE STATISTICS join_mcv_bad_wholerow (mcv) ON k.keyword +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk = k); +-- a subquery in the FROM clause +CREATE STATISTICS join_mcv_bad_subquery (mcv) ON x.keyword +FROM (SELECT * FROM join_mcv_keywords) x + JOIN join_mcv_movies mk ON (x.id = mk.keyword_id); +-- a sampled table +CREATE STATISTICS join_mcv_bad_tablesample (mcv) ON k.keyword +FROM join_mcv_movies mk TABLESAMPLE system (10) + JOIN join_mcv_keywords k ON (mk.keyword_id = k.id); +-- a virtual generated column +CREATE TABLE join_mcv_vgc (id INTEGER PRIMARY KEY, v INTEGER, + g INTEGER GENERATED ALWAYS AS (v * 2) VIRTUAL); +CREATE STATISTICS join_mcv_bad_vgc (mcv) ON v.g +FROM join_mcv_movies mk JOIN join_mcv_vgc v ON (mk.keyword_id = v.id); +DROP TABLE join_mcv_vgc; +-- a column of a type without a less-than operator, even alone, as the MCV +-- list is built by sorting +CREATE TABLE join_mcv_xid (id INTEGER PRIMARY KEY, w xid); +CREATE STATISTICS join_mcv_xid_stats (mcv) ON x.w +FROM join_mcv_movies mk JOIN join_mcv_xid x ON (mk.keyword_id = x.id); +CREATE STATISTICS join_mcv_xid_multi_stats (mcv) ON x.w, mk.keyword_id +FROM join_mcv_movies mk JOIN join_mcv_xid x ON (mk.keyword_id = x.id); +DROP TABLE join_mcv_xid; +-- a partitioned table, whichever of the tables it is +CREATE TABLE join_mcv_plain (id int); +CREATE TABLE join_mcv_partitioned (id int, val int) PARTITION BY RANGE (id); +CREATE TABLE join_mcv_partitioned_1 PARTITION OF join_mcv_partitioned + FOR VALUES FROM (0) TO (100); +CREATE TABLE join_mcv_partitioned_2 PARTITION OF join_mcv_partitioned + FOR VALUES FROM (100) TO (200); +CREATE INDEX ON join_mcv_partitioned (id); +CREATE STATISTICS join_mcv_part_probed_stats (mcv) ON p.val +FROM join_mcv_plain s JOIN join_mcv_partitioned p ON (s.id = p.id); +CREATE INDEX ON join_mcv_plain (id); +CREATE STATISTICS join_mcv_part_anchor_stats (mcv) ON s.id +FROM join_mcv_partitioned p JOIN join_mcv_plain s ON (p.id = s.id); +DROP TABLE join_mcv_plain, join_mcv_partitioned; + +-- USING is another way of writing an equality join condition +CREATE TABLE join_mcv_keyword_notes (keyword_id INTEGER PRIMARY KEY, note TEXT) + WITH (autovacuum_enabled = off); +INSERT INTO join_mcv_keyword_notes SELECT id, 'note' FROM join_mcv_keywords; +ANALYZE join_mcv_keyword_notes; +CREATE STATISTICS join_mcv_using_stats (mcv) ON n.note +FROM join_mcv_movies mk JOIN join_mcv_keyword_notes n USING (keyword_id); +SELECT pg_get_statisticsobjdef(oid, false) FROM pg_statistic_ext +WHERE stxname = 'join_mcv_using_stats'; +DROP STATISTICS join_mcv_using_stats; +-- a join of three tables is accepted, with a column from each joined table +-- printed with its own table and the tables stored in the order written; +-- ANALYZE warns that such statistics are not built yet +CREATE STATISTICS join_mcv_three_tables_stats (mcv) ON k.keyword, n.note + FROM join_mcv_movies mk + JOIN join_mcv_keywords k ON (mk.keyword_id = k.id) + JOIN join_mcv_keyword_notes n ON (k.id = n.keyword_id); +SELECT stxjoinrels::regclass[], pg_get_statisticsobjdef(oid, false) +FROM pg_statistic_ext WHERE stxname = 'join_mcv_three_tables_stats'; +ANALYZE join_mcv_movies; +DROP STATISTICS join_mcv_three_tables_stats; +-- the same join parenthesized: the tables are stored in an order in which +-- each one is joined to an earlier one, here the notes before the keywords +CREATE STATISTICS join_mcv_nested_stats (mcv) ON k.keyword, n.note + FROM join_mcv_movies mk + JOIN (join_mcv_keywords k + JOIN join_mcv_keyword_notes n ON (k.id = n.keyword_id)) + ON (mk.keyword_id = n.keyword_id); +SELECT stxjoinrels::regclass[], pg_get_statisticsobjdef(oid, false) +FROM pg_statistic_ext WHERE stxname = 'join_mcv_nested_stats'; +DROP STATISTICS join_mcv_nested_stats; +-- a join condition may only refer to tables joined so far +CREATE STATISTICS join_mcv_bad_forward_ref (mcv) ON k.keyword, n.note + FROM join_mcv_movies mk + JOIN join_mcv_keywords k ON (mk.keyword_id = n.keyword_id) + JOIN join_mcv_keyword_notes n ON (k.id = n.keyword_id); +-- every table must be connected to the others by a join condition +CREATE STATISTICS join_mcv_bad_unconnected (mcv) ON k.keyword, n.note + FROM join_mcv_movies mk + JOIN join_mcv_keywords k ON (mk.keyword_id = k.id) + JOIN join_mcv_keyword_notes n ON (mk.movie_id = k.id); +-- several conditions between the same two tables are accepted, but not +-- built yet either: ANALYZE skips them, and leaves statistics already in +-- place, such as restored ones +CREATE STATISTICS join_mcv_multiclause_stats (mcv) ON k.keyword +FROM join_mcv_movies mk + JOIN join_mcv_keywords k ON (mk.keyword_id = k.id AND mk.movie_id = k.id); +SELECT pg_get_statisticsobjdef(oid, false) FROM pg_statistic_ext +WHERE stxname = 'join_mcv_multiclause_stats'; +SELECT pg_restore_extended_stats( + 'schemaname', 'public', + 'relname', 'join_mcv_movies', + 'statistics_schemaname', 'public', + 'statistics_name', 'join_mcv_multiclause_stats', + 'inherited', false, + 'most_common_vals', '{{x}}'::text[], + 'most_common_freqs', '{1}'::double precision[], + 'most_common_base_freqs', '{1}'::double precision[]); +SELECT most_common_vals FROM pg_stats_ext +WHERE statistics_name = 'join_mcv_multiclause_stats'; +ANALYZE join_mcv_movies; +SELECT most_common_vals FROM pg_stats_ext +WHERE statistics_name = 'join_mcv_multiclause_stats'; +DROP STATISTICS join_mcv_multiclause_stats; +-- without a list of kinds, join statistics get every kind they support, +-- which is only mcv, whatever the number of columns +CREATE STATISTICS join_mcv_kinds_stats_1 ON k.keyword +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword_id = k.id); +CREATE STATISTICS join_mcv_kinds_stats_2 ON k.keyword, mk.year +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword_id = k.id); +SELECT stxname, stxkind FROM pg_statistic_ext +WHERE stxname LIKE 'join_mcv_kinds%' ORDER BY stxname; +DROP STATISTICS join_mcv_kinds_stats_1, join_mcv_kinds_stats_2; + +-- check statistics are dropped when a column of theirs is, whichever table +-- it belongs to, and when a join column is +CREATE STATISTICS join_mcv_dep_note_stats (mcv) ON n.note +FROM join_mcv_movies mk JOIN join_mcv_keyword_notes n USING (keyword_id); +CREATE STATISTICS join_mcv_dep_year_stats (mcv) ON mk.year +FROM join_mcv_movies mk JOIN join_mcv_keyword_notes n USING (keyword_id); +ALTER TABLE join_mcv_keyword_notes DROP COLUMN note; +\d join_mcv_movies +ALTER TABLE join_mcv_keyword_notes DROP COLUMN keyword_id; +\d join_mcv_movies +DROP TABLE join_mcv_keyword_notes; + +-- check dropping a table in the middle of a join of 60 tables drops the +-- statistics (the statements are long, so they are not echoed) +\set ECHO none +SELECT format('CREATE TABLE join_many_%s (k int); CREATE INDEX ON join_many_%s (k);', i, i) +FROM generate_series(1, 60) i +\gexec +SELECT 'CREATE STATISTICS join_many_stats (mcv) ON t1.k, t60.k FROM join_many_1 t1 ' || + string_agg(format('JOIN join_many_%s t%s ON (t%s.k = t%s.k)', i, i, i - 1, i), + ' ' ORDER BY i) +FROM generate_series(2, 60) i +\gexec +\set ECHO all +SELECT stxname FROM pg_statistic_ext WHERE stxname = 'join_many_stats'; +DROP TABLE join_many_30; +SELECT stxname FROM pg_statistic_ext WHERE stxname = 'join_many_stats'; +\set ECHO none +SELECT format('DROP TABLE join_many_%s', i) +FROM generate_series(1, 60) i WHERE i <> 30 +\gexec +\set ECHO all + +-- check the frequencies are fractions of the joined rows, scaled by the +-- size of the join: val = 'hit' is most of the probed table but few of the +-- joined rows, nearly all of which join id 10 +CREATE TABLE join_mcv_scale (id int) WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_scale_indexed (id int, val text) + WITH (autovacuum_enabled = off); +INSERT INTO join_mcv_scale +SELECT CASE WHEN g <= 9 THEN g ELSE 10 END FROM generate_series(1, 1000) g; +INSERT INTO join_mcv_scale_indexed +SELECT g, CASE WHEN g <= 9 THEN 'hit' ELSE 'miss' END +FROM generate_series(1, 10) g; +CREATE INDEX ON join_mcv_scale_indexed (id); +ANALYZE join_mcv_scale, join_mcv_scale_indexed; + +-- Without join statistics: the filter and the join are treated as +-- independent +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_scale s + JOIN join_mcv_scale_indexed i ON (s.id = i.id) + WHERE i.val = ''hit'''); + +-- With join statistics +CREATE STATISTICS join_mcv_scale_stats (mcv) ON i.val +FROM join_mcv_scale s JOIN join_mcv_scale_indexed i ON (s.id = i.id); +ANALYZE join_mcv_scale; +SELECT m.values, m.frequency, m.base_frequency +FROM pg_statistic_ext s +JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +CROSS JOIN LATERAL pg_mcv_list_items(d.stxdmcv) AS m +WHERE s.stxname = 'join_mcv_scale_stats' +ORDER BY m.frequency DESC; +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_scale s + JOIN join_mcv_scale_indexed i ON (s.id = i.id) + WHERE i.val = ''hit'''); + +DROP TABLE join_mcv_scale, join_mcv_scale_indexed; + +-- check statistics of two tables are used when a third table is joined to +-- either of them, in any order; the third table has one row per id, so it +-- changes neither the rows of the join nor the filter +CREATE TABLE join_mcv_three (id int) WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_three_indexed (id int, y int) + WITH (autovacuum_enabled = off); +CREATE INDEX ON join_mcv_three_indexed (id); +CREATE TABLE join_mcv_three_other (id int) WITH (autovacuum_enabled = off); + +-- y = 1 in a few rows +INSERT INTO join_mcv_three_indexed +SELECT g, (g <= 10)::int FROM generate_series(1, 100) g; + +-- most rows join the rows where y = 1 +INSERT INTO join_mcv_three +SELECT CASE WHEN g <= 900 THEN 1 + g % 10 ELSE 11 + g % 90 END +FROM generate_series(1, 1000) g; + +-- 100 rows, one for each id +INSERT INTO join_mcv_three_other SELECT g FROM generate_series(1, 100) g; + +ANALYZE join_mcv_three, join_mcv_three_indexed, join_mcv_three_other; + +-- Without join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_three_other o + JOIN join_mcv_three f ON (o.id = f.id) + JOIN join_mcv_three_indexed i ON (f.id = i.id) + WHERE i.y = 1'); + +CREATE STATISTICS join_mcv_three_stats (mcv) ON i.y +FROM join_mcv_three f JOIN join_mcv_three_indexed i ON (f.id = i.id); +ANALYZE join_mcv_three; + +-- The two tables of the statistics alone +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_three f + JOIN join_mcv_three_indexed i ON (f.id = i.id) + WHERE i.y = 1'); + +-- With the third table joined to either table of the statistics: the +-- estimate is the same whichever pair the planner joins first +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_three_other o + JOIN join_mcv_three f ON (o.id = f.id) + JOIN join_mcv_three_indexed i ON (f.id = i.id) + WHERE i.y = 1'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_three_other o + JOIN join_mcv_three_indexed i ON (o.id = i.id) + JOIN join_mcv_three f ON (f.id = i.id) + WHERE i.y = 1'); + +DROP TABLE join_mcv_three, join_mcv_three_indexed, join_mcv_three_other; + +-- check statistics correct a foreign key's estimate for a filter: the +-- foreign key does not account for NULL keys, and that error stays +CREATE TABLE join_mcv_fk_null_indexed (id int PRIMARY KEY, y int) + WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_fk_null (id int REFERENCES join_mcv_fk_null_indexed (id)) + WITH (autovacuum_enabled = off); + +-- y = 1 in a few rows +INSERT INTO join_mcv_fk_null_indexed +SELECT g, (g <= 10)::int FROM generate_series(1, 100) g; + +-- some rows reference no row, and most of the others reference the rows +-- where y = 1 +INSERT INTO join_mcv_fk_null +SELECT CASE WHEN g <= 200 THEN NULL + WHEN g <= 920 THEN 1 + g % 10 + ELSE 11 + g % 80 END +FROM generate_series(1, 1000) g; + +ANALYZE join_mcv_fk_null, join_mcv_fk_null_indexed; + +-- Without join statistics, with and without a filter +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_fk_null s + JOIN join_mcv_fk_null_indexed i ON (s.id = i.id)'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_fk_null s + JOIN join_mcv_fk_null_indexed i ON (s.id = i.id) + WHERE i.y = 1'); + +-- With join statistics, which correct the foreign key's estimate for the +-- filter only: the remaining error is the foreign key's own, as it does not +-- account for the NULL keys +CREATE STATISTICS join_mcv_fk_null_stats (mcv) ON i.y +FROM join_mcv_fk_null s JOIN join_mcv_fk_null_indexed i ON (s.id = i.id); +ANALYZE join_mcv_fk_null; +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_fk_null s + JOIN join_mcv_fk_null_indexed i ON (s.id = i.id) + WHERE i.y = 1'); + +DROP TABLE join_mcv_fk_null, join_mcv_fk_null_indexed; + +-- check estimates for values stored in the join statistics, and for values +-- not stored there +CREATE TABLE join_mcv_stored_indexed (id int PRIMARY KEY, b int, c int) + WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_stored (dim_id int, a int) + WITH (autovacuum_enabled = off); + +-- common values of b, single rows, and a value no row joins +INSERT INTO join_mcv_stored_indexed +SELECT i, CASE WHEN i <= 30 THEN 1 WHEN i <= 50 THEN 2 WHEN i = 51 THEN 3 + WHEN i = 52 THEN 4 WHEN i <= 72 THEN 5 ELSE 6 END, + 1 + i % 4 +FROM generate_series(1, 80) i; + +-- a is never below c +INSERT INTO join_mcv_stored +SELECT dim.id, dim.c + (g % 3 = 0)::int +FROM (VALUES (1, 30, 70), (31, 20, 50), (51, 1, 25), (52, 1, 15), (53, 20, 6)) + AS v (first_id, nrows, njoined), + generate_series(1, v.njoined) g, + join_mcv_stored_indexed dim +WHERE dim.id = v.first_id + g % v.nrows; + +-- 10 rows that join nothing +INSERT INTO join_mcv_stored SELECT NULL, 1 FROM generate_series(1, 10); + +ANALYZE join_mcv_stored_indexed, join_mcv_stored; + +-- Without join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b = 1'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b IN (1, 3)'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b = 3'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b = 6'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b > 2'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE f.a = 2 AND dim.c = 2'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.c = 4'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE f.a > 2 AND dim.c > 2'); + +-- With join statistics that store every value of b and every (a, c) +-- combination in the join +CREATE STATISTICS join_mcv_stored_b_stats (mcv) ON dim.b +FROM join_mcv_stored f JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id); +CREATE STATISTICS join_mcv_stored_a_c_stats (mcv) ON f.a, dim.c +FROM join_mcv_stored f JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id); +ANALYZE join_mcv_stored; +SELECT s.stxname, m.values, round(m.frequency::numeric, 3) AS frequency +FROM pg_statistic_ext s +JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +CROSS JOIN LATERAL pg_mcv_list_items(d.stxdmcv) AS m +WHERE s.stxname IN ('join_mcv_stored_b_stats', 'join_mcv_stored_a_c_stats') +ORDER BY s.stxname, m.frequency DESC; +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b = 1'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b IN (1, 3)'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b = 3'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b = 6'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b > 2'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE f.a = 2 AND dim.c = 2'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.c = 4'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE f.a > 2 AND dim.c > 2'); + +-- With lower statistics targets, only the most common values are stored; +-- the joined rows all fit in the sample, so the statistics are the same on +-- every run +ALTER STATISTICS join_mcv_stored_b_stats SET STATISTICS 2; +ALTER STATISTICS join_mcv_stored_a_c_stats SET STATISTICS 3; +ANALYZE join_mcv_stored; +SELECT s.stxname, m.values, round(m.frequency::numeric, 3) AS frequency +FROM pg_statistic_ext s +JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +CROSS JOIN LATERAL pg_mcv_list_items(d.stxdmcv) AS m +WHERE s.stxname IN ('join_mcv_stored_b_stats', 'join_mcv_stored_a_c_stats') +ORDER BY s.stxname, m.frequency DESC; +-- A stored value. +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b = 1'); +-- One value stored and one not: the frequency of b = 1, plus the estimate +-- without join statistics for b = 3 +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b IN (1, 3)'); +-- A value not stored gets the estimate without join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b = 3'); +-- The statistics no longer show that b = 6 joins nothing, so the estimate +-- without join statistics is kept +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b = 6'); +-- A range no stored value satisfies: the estimate is capped at the joined +-- rows of the values not stored +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b > 2'); +-- A combination not stored gets the estimate without join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE f.a = 2 AND dim.c = 2'); +-- (4, 4) is stored; the other combinations with c = 4 get the estimate +-- without join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE dim.c = 4'); +-- A range: (3, 3) and (4, 4) are stored, and the other combinations it +-- selects get the estimate without join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_stored f + JOIN join_mcv_stored_indexed dim ON (f.dim_id = dim.id) + WHERE f.a > 2 AND dim.c > 2'); + +-- Leave join_mcv_stored, join_mcv_stored_indexed and their statistics in +-- place for pg_upgrade testing. + +-- check a value stored at a frequency much lower in the join than in its +-- table: b = 0 is 30 percent of the dimension rows but 1 percent of the +-- joined rows, the most common value there +CREATE TABLE join_mcv_diluted (dim_id int) WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_diluted_indexed (id int PRIMARY KEY, b int) + WITH (autovacuum_enabled = off); + +-- b = 0 in many rows, and a distinct b in each other row +INSERT INTO join_mcv_diluted_indexed +SELECT g, CASE WHEN g <= 90 THEN 0 ELSE g END FROM generate_series(1, 300) g; + +-- few rows reference the rows where b = 0; the others reference more values +-- of b than the statistics store +INSERT INTO join_mcv_diluted +SELECT CASE WHEN g <= 10 THEN g ELSE 91 + g % 210 END +FROM generate_series(1, 1000) g; + +ANALYZE join_mcv_diluted, join_mcv_diluted_indexed; + +-- Without join statistics: b = 0 is 30 percent of the dimension table +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_diluted f + JOIN join_mcv_diluted_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b = 0'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_diluted f + JOIN join_mcv_diluted_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b < 1'); + +-- With join statistics, which store b = 0 at 1 percent of the joined rows +CREATE STATISTICS join_mcv_diluted_stats (mcv) ON dim.b +FROM join_mcv_diluted f + JOIN join_mcv_diluted_indexed dim ON (f.dim_id = dim.id); +ANALYZE join_mcv_diluted; +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_diluted f + JOIN join_mcv_diluted_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b = 0'); +-- A range selecting that value alone is measured the same way +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_diluted f + JOIN join_mcv_diluted_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b < 1'); + +DROP TABLE join_mcv_diluted, join_mcv_diluted_indexed; + +-- check a filter on the anchor table expected to select less than one row +-- is corrected as selecting one row, as a filter on the probed table is, +-- since the planner estimates at least one row for a table +CREATE TABLE join_mcv_rare (id int, a int) WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_rare_indexed (id int) WITH (autovacuum_enabled = off); +CREATE INDEX ON join_mcv_rare_indexed (id); + +-- half the rows have id 1 +INSERT INTO join_mcv_rare_indexed +SELECT CASE WHEN g <= 500 THEN 1 ELSE g - 499 END +FROM generate_series(1, 1000) g; + +-- the one row with a = -10 references id 1; the others, with a = 0, +-- reference one row each +INSERT INTO join_mcv_rare +SELECT CASE WHEN g = 1 THEN 1 ELSE 2 + g % 500 END, + CASE WHEN g = 1 THEN -10 ELSE 0 END +FROM generate_series(1, 1000) g; + +ANALYZE join_mcv_rare, join_mcv_rare_indexed; + +-- Without join statistics: a = -10, and a < -5 with no histogram to +-- estimate it from, are each expected to select one row +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_rare s + JOIN join_mcv_rare_indexed i ON (s.id = i.id) + WHERE s.a = -10'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_rare s + JOIN join_mcv_rare_indexed i ON (s.id = i.id) + WHERE s.a < -5'); + +-- With join statistics, which store that a third of the joined rows have +-- a = -10: the two filters get the same estimate +CREATE STATISTICS join_mcv_rare_stats (mcv) ON s.a +FROM join_mcv_rare s JOIN join_mcv_rare_indexed i ON (s.id = i.id); +ANALYZE join_mcv_rare; +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_rare s + JOIN join_mcv_rare_indexed i ON (s.id = i.id) + WHERE s.a = -10'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_rare s + JOIN join_mcv_rare_indexed i ON (s.id = i.id) + WHERE s.a < -5'); + +DROP TABLE join_mcv_rare, join_mcv_rare_indexed; + +-- check which filter clauses the join statistics can estimate +CREATE TABLE join_mcv_filters_indexed (id int PRIMARY KEY, b int, c int, d bool) + WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_filters (dim_id int, a int) + WITH (autovacuum_enabled = off); + +-- c NULL in some rows, d true in fewer +INSERT INTO join_mcv_filters_indexed +SELECT i, 1 + i % 4, NULLIF(i % 5, 0), i % 10 = 0 +FROM generate_series(1, 40) i; + +-- a third of the rows join the rows where c is NULL; a is never below b +INSERT INTO join_mcv_filters +SELECT dim.id, dim.b + (g % 3 = 0)::int +FROM generate_series(1, 600) g +JOIN join_mcv_filters_indexed dim + ON dim.id = CASE WHEN g <= 200 THEN 5 * (1 + g % 8) ELSE 1 + g % 40 END; + +ANALYZE join_mcv_filters_indexed, join_mcv_filters; + +-- Without join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE f.a > 2 AND dim.b > 2'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE 2 < f.a AND 2 < dim.b'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE f.a <> 1 AND dim.b <> 1'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE f.a IN (1, 2) AND dim.b = 1'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE f.a IN (1, NULL, 2) AND dim.b = 1'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE f.a <= ANY (ARRAY[1, 2]) AND dim.b = 1'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE f.a < ALL (ARRAY[3, 4]) AND dim.b = 2'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE dim.c NOT IN (1, 2)'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE (f.a = 1 OR f.a = 5) AND dim.b = 4'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE (dim.c = 1 OR dim.c IS NULL)'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE dim.c IS NULL'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE dim.c IS NOT NULL'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE dim.d'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE NOT dim.d'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE dim.d IS TRUE'); + +-- With join statistics that store every combination in the join +CREATE STATISTICS join_mcv_filters_stats (mcv) ON f.a, dim.b, dim.c, dim.d +FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id); +ANALYZE join_mcv_filters; +SELECT count(*) AS items, round(sum(m.frequency)::numeric, 2) AS total_frequency +FROM pg_statistic_ext s +JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +CROSS JOIN LATERAL pg_mcv_list_items(d.stxdmcv) AS m +WHERE s.stxname = 'join_mcv_filters_stats'; +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE f.a > 2 AND dim.b > 2'); +-- A constant on the left: the same as "f.a > 2 AND dim.b > 2". +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE 2 < f.a AND 2 < dim.b'); +-- "<>" on both columns. +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE f.a <> 1 AND dim.b <> 1'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE f.a IN (1, 2) AND dim.b = 1'); +-- NULL array elements are dropped, so this is the same as "f.a IN (1, 2)". +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE f.a IN (1, NULL, 2) AND dim.b = 1'); +-- An inequality over an array: the same as "f.a IN (1, 2)" here. +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE f.a <= ANY (ARRAY[1, 2]) AND dim.b = 1'); +-- ALL: the same as "f.a < 3". +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE f.a < ALL (ARRAY[3, 4]) AND dim.b = 2'); +-- NOT IN is "<> ALL". +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE dim.c NOT IN (1, 2)'); +-- OR of filters on one table. +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE (f.a = 1 OR f.a = 5) AND dim.b = 4'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE (dim.c = 1 OR dim.c IS NULL)'); +-- IS [NOT] NULL. +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE dim.c IS NULL'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE dim.c IS NOT NULL'); +-- A boolean column, alone and negated. +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE dim.d'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE NOT dim.d'); +-- IS TRUE is not a form the statistics match: estimated as without them. +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_filters f + JOIN join_mcv_filters_indexed dim ON (f.dim_id = dim.id) + WHERE dim.d IS TRUE'); + +-- Leave join_mcv_filters, join_mcv_filters_indexed and their statistics in +-- place for pg_upgrade testing. + +-- check a filter the join statistics do not cover is estimated from the +-- probed table's own statistics: color and shape always go together, and +-- the join statistics cover color only +CREATE TABLE join_mcv_partial_indexed ( + id INTEGER PRIMARY KEY, + color TEXT NOT NULL, + shape TEXT NOT NULL +) WITH (autovacuum_enabled = off); + +CREATE TABLE join_mcv_partial ( + id INTEGER PRIMARY KEY, + dim_id INTEGER NOT NULL +) WITH (autovacuum_enabled = off); + +-- each color with its own shape +INSERT INTO join_mcv_partial_indexed (id, color, shape) +SELECT i, + CASE (i - 1) / 20 + WHEN 0 THEN 'red' + WHEN 1 THEN 'blue' + WHEN 2 THEN 'green' + WHEN 3 THEN 'yellow' + ELSE 'white' + END, + CASE (i - 1) / 20 + WHEN 0 THEN 'circle' + WHEN 1 THEN 'square' + WHEN 2 THEN 'triangle' + WHEN 3 THEN 'star' + ELSE 'diamond' + END +FROM generate_series(1, 100) i; + +-- half the rows reference the red rows, none the yellow or white ones +INSERT INTO join_mcv_partial (id, dim_id) +SELECT i, + CASE + WHEN i % 100 < 50 THEN (i % 20) + 1 + WHEN i % 100 < 80 THEN (i % 20) + 21 + ELSE (i % 20) + 41 + END +FROM generate_series(1, 10000) i; + +ANALYZE join_mcv_partial_indexed; +ANALYZE join_mcv_partial; + +-- Single-table statistics on the probed table know that color and shape go +-- together +CREATE STATISTICS join_mcv_partial_single_stats (mcv) ON color, shape +FROM join_mcv_partial_indexed; +ANALYZE join_mcv_partial_indexed; + +-- Without join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_partial f, join_mcv_partial_indexed d + WHERE d.color = ''red'' AND d.shape = ''circle'' + AND f.dim_id = d.id +'); + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_partial f, join_mcv_partial_indexed d + WHERE d.color = ''red'' AND d.shape = ''square'' + AND f.dim_id = d.id +'); + +-- With join statistics covering color only +CREATE STATISTICS join_mcv_partial_stats (mcv) ON d.color +FROM join_mcv_partial f JOIN join_mcv_partial_indexed d ON (f.dim_id = d.id); +ANALYZE join_mcv_partial; + +-- red and circle go together: the statistics measure the pair +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_partial f, join_mcv_partial_indexed d + WHERE d.color = ''red'' AND d.shape = ''circle'' + AND f.dim_id = d.id +'); + +-- red and square never go together: the estimate is the one without join +-- statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_partial f, join_mcv_partial_indexed d + WHERE d.color = ''red'' AND d.shape = ''square'' + AND f.dim_id = d.id +'); + +DROP TABLE join_mcv_partial, join_mcv_partial_indexed CASCADE; + +-- check two statistics objects are used in one join, one for each +-- dimension table of a fact table +CREATE TABLE join_mcv_star ( + id INTEGER PRIMARY KEY, + dim1_id INTEGER NOT NULL, + dim2_id INTEGER NOT NULL +) WITH (autovacuum_enabled = off); + +CREATE TABLE join_mcv_star_dim1 ( + id INTEGER PRIMARY KEY, + label TEXT NOT NULL +) WITH (autovacuum_enabled = off); + +CREATE TABLE join_mcv_star_dim2 ( + id INTEGER PRIMARY KEY, + color TEXT NOT NULL +) WITH (autovacuum_enabled = off); + +-- join_mcv_star_dim1: 50 rows +INSERT INTO join_mcv_star_dim1 (id, label) +SELECT i, 'label_' || i FROM generate_series(1, 50) i; + +-- join_mcv_star_dim2: 20 rows +INSERT INTO join_mcv_star_dim2 (id, color) +SELECT i, 'color_' || i FROM generate_series(1, 20) i; + +-- skewed to both dimensions: dim1_id mostly among the first ten ids, +-- dim2_id mostly id 1, and some rows reference no dim2 row +INSERT INTO join_mcv_star (id, dim1_id, dim2_id) +SELECT i, + CASE + WHEN i % 100 < 60 THEN (i % 10) + 1 + WHEN i % 100 < 90 THEN (i % 10) + 11 + ELSE (i % 10) + 21 + END, + CASE + WHEN i % 100 < 50 THEN 1 + WHEN i % 100 < 80 THEN 2 + ELSE (i % 20) + 3 + END +FROM generate_series(1, 10000) i; + +ANALYZE join_mcv_star; +ANALYZE join_mcv_star_dim1; +ANALYZE join_mcv_star_dim2; + +-- Without join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_star f + JOIN join_mcv_star_dim1 d1 ON (f.dim1_id = d1.id) + JOIN join_mcv_star_dim2 d2 ON (f.dim2_id = d2.id) + WHERE d1.label = ''label_1'' AND d2.color = ''color_1'' +'); + +-- With join statistics for each pair +CREATE STATISTICS join_mcv_star_dim1_stats (mcv) ON d1.label +FROM join_mcv_star f JOIN join_mcv_star_dim1 d1 ON (f.dim1_id = d1.id); +ANALYZE join_mcv_star; + +CREATE STATISTICS join_mcv_star_dim2_stats (mcv) ON d2.color +FROM join_mcv_star f JOIN join_mcv_star_dim2 d2 ON (f.dim2_id = d2.id); +ANALYZE join_mcv_star; + +-- Three-way join with both objects applicable: label_1 and color_1 go +-- together, which neither object knows, so the two corrections combine as +-- independent +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_star f + JOIN join_mcv_star_dim1 d1 ON (f.dim1_id = d1.id) + JOIN join_mcv_star_dim2 d2 ON (f.dim2_id = d2.id) + WHERE d1.label = ''label_1'' AND d2.color = ''color_1'' +'); + +-- Dropping the color object: label_1 is still measured, color_1 is +-- estimated as 1 color in 20 +DROP STATISTICS join_mcv_star_dim2_stats; + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_star f + JOIN join_mcv_star_dim1 d1 ON (f.dim1_id = d1.id) + JOIN join_mcv_star_dim2 d2 ON (f.dim2_id = d2.id) + WHERE d1.label = ''label_1'' AND d2.color = ''color_1'' +'); + +-- No statistics at all +DROP STATISTICS join_mcv_star_dim1_stats; + +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_star f + JOIN join_mcv_star_dim1 d1 ON (f.dim1_id = d1.id) + JOIN join_mcv_star_dim2 d2 ON (f.dim2_id = d2.id) + WHERE d1.label = ''label_1'' AND d2.color = ''color_1'' +'); + +DROP TABLE join_mcv_star, join_mcv_star_dim1, join_mcv_star_dim2; + +-- check an inequality join is not the condition the statistics were built +-- on, so the estimate is the one without statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_movies mk, join_mcv_keywords k + WHERE k.keyword = ''keyword_1'' AND mk.keyword_id > k.id +'); +DROP STATISTICS join_mcv_movies_keyword_stats; +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_movies mk, join_mcv_keywords k + WHERE k.keyword = ''keyword_1'' AND mk.keyword_id > k.id +'); + +-- check join statistics are not used for a join of their tables on more +-- than one condition, and what counts as a condition +CREATE TABLE join_mcv_two (a int, b int) WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_two_indexed (a int, b int, y int) + WITH (autovacuum_enabled = off); +CREATE INDEX ON join_mcv_two_indexed (a); +CREATE INDEX ON join_mcv_two_indexed (b); + +-- every pair (a, b) once, with y = 1 where a or b is 1 +INSERT INTO join_mcv_two_indexed +SELECT a, b, (a = 1 OR b = 1)::int +FROM generate_series(1, 10) a, generate_series(1, 10) b; + +-- most rows join (1, 1), where y = 1 +INSERT INTO join_mcv_two +SELECT CASE WHEN g <= 900 THEN 1 ELSE 2 + g % 9 END, + CASE WHEN g <= 900 THEN 1 ELSE 2 + (g / 9) % 9 END +FROM generate_series(1, 1000) g; + +ANALYZE join_mcv_two, join_mcv_two_indexed; + +-- Without join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_two s + JOIN join_mcv_two_indexed i ON (s.a = i.a AND s.b = i.b) + WHERE i.y = 1'); + +-- With join statistics for one of the conditions +CREATE STATISTICS join_mcv_two_a_stats (mcv) ON i.y +FROM join_mcv_two s JOIN join_mcv_two_indexed i ON (s.a = i.a); +ANALYZE join_mcv_two; +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_two s + JOIN join_mcv_two_indexed i ON (s.a = i.a AND s.b = i.b) + WHERE i.y = 1'); + +-- A condition equal to a constant becomes a restriction on each table rather +-- than a join condition, so the join statistics are used +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_two s + JOIN join_mcv_two_indexed i ON (s.a = i.a AND s.b = i.b) + WHERE i.y = 1 AND i.b = 1'); + +-- Expressions over both tables, with join statistics for one of the +-- conditions. An expression that is only sorted on is not a join +-- condition, so the join statistics are used +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_two s + JOIN join_mcv_two_indexed i ON (s.a = i.a) + WHERE i.y = 1 + ORDER BY s.b + i.b'); + +-- An expression equal to the join key is a second join condition, so the +-- join statistics are not used +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_two s + JOIN join_mcv_two_indexed i ON (s.a = i.a AND i.a = s.b + i.b - 1) + WHERE i.y = 1'); + +-- Outer joins, with join statistics for one of the conditions +CREATE TABLE join_mcv_third (k int) WITH (autovacuum_enabled = off); +INSERT INTO join_mcv_third SELECT g FROM generate_series(1, 1000) g; +ANALYZE join_mcv_third; + +-- A filter on the anchor table in the join condition of a left join is not a +-- join condition, so the join statistics are used +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_two s + LEFT JOIN join_mcv_two_indexed i ON (s.a = i.a AND i.y = 1 AND s.b = 1)'); + +-- A condition on both tables above a left join is a second join condition, +-- so the join statistics are not used +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_two s + LEFT JOIN join_mcv_two_indexed i ON (s.a = i.a AND i.y = 1) + WHERE (i.b = s.b OR i.b IS NULL)'); + +-- One condition, on a column nullable by a full join below, so the join +-- statistics are used +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_third t + FULL JOIN join_mcv_two s ON (t.k = s.a) + LEFT JOIN join_mcv_two_indexed i ON (s.a = i.a AND i.y = 1)'); + +-- The same with a left join below, for which the planner keeps a copy of +-- the join clause for each join order +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_third t + LEFT JOIN join_mcv_two s ON (t.k = s.a) + LEFT JOIN join_mcv_two_indexed i ON (s.a = i.a AND i.y = 1)'); + +DROP TABLE join_mcv_two, join_mcv_two_indexed, join_mcv_third; + +-- check a foreign key with another condition counts as two conditions +CREATE TABLE join_mcv_fk_indexed (id int PRIMARY KEY, b int, y int) + WITH (autovacuum_enabled = off); +CREATE INDEX ON join_mcv_fk_indexed (b); +CREATE TABLE join_mcv_fk (id int REFERENCES join_mcv_fk_indexed (id), b int) + WITH (autovacuum_enabled = off); + +-- groups of ten rows by b, with y = 1 only in the group b = 0 +INSERT INTO join_mcv_fk_indexed +SELECT g, (g - 1) / 10, (g <= 10)::int FROM generate_series(1, 200) g; + +-- few rows have the b of the row they reference +INSERT INTO join_mcv_fk +SELECT g % 10 + 1, (g / 10) % 20 FROM generate_series(0, 999) g; +INSERT INTO join_mcv_fk +SELECT i.id, CASE WHEN g % 20 = 0 THEN i.b ELSE 0 END +FROM generate_series(0, 999) g +JOIN join_mcv_fk_indexed i ON (i.id = 11 + g % 190); + +ANALYZE join_mcv_fk, join_mcv_fk_indexed; + +-- Without join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_fk s + JOIN join_mcv_fk_indexed i ON (s.id = i.id AND s.b = i.b) + WHERE i.y = 1'); + +-- With join statistics for the foreign key: the estimate is unchanged +CREATE STATISTICS join_mcv_fk_id_stats (mcv) ON i.y +FROM join_mcv_fk s JOIN join_mcv_fk_indexed i ON (s.id = i.id); +ANALYZE join_mcv_fk; +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_fk s + JOIN join_mcv_fk_indexed i ON (s.id = i.id AND s.b = i.b) + WHERE i.y = 1'); + +DROP TABLE join_mcv_fk, join_mcv_fk_indexed; + +-- check two foreign keys count as two conditions, and the error is in the +-- planner's estimate of the join itself, which no correction can repair +CREATE TABLE join_mcv_fk2_indexed (id int PRIMARY KEY, code int UNIQUE, y int) + WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_fk2 (id int REFERENCES join_mcv_fk2_indexed (id), + code int REFERENCES join_mcv_fk2_indexed (code)) + WITH (autovacuum_enabled = off); + +-- code = id, with y = 1 in a few rows +INSERT INTO join_mcv_fk2_indexed +SELECT g, g, (g <= 10)::int FROM generate_series(1, 100) g; + +-- half the rows reference the rows where y = 1, by both keys +INSERT INTO join_mcv_fk2 +SELECT r, r +FROM (SELECT CASE WHEN g <= 500 THEN 1 + g % 10 ELSE 11 + g % 90 END + FROM generate_series(1, 1000) g) AS s(r); + +ANALYZE join_mcv_fk2, join_mcv_fk2_indexed; + +-- Without join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_fk2 s + JOIN join_mcv_fk2_indexed i ON (s.id = i.id AND s.code = i.code) + WHERE i.y = 1'); + +-- With join statistics for each of the foreign keys: neither applies +CREATE STATISTICS join_mcv_fk2_id_stats (mcv) ON i.y +FROM join_mcv_fk2 s JOIN join_mcv_fk2_indexed i ON (s.id = i.id); +CREATE STATISTICS join_mcv_fk2_code_stats (mcv) ON i.y +FROM join_mcv_fk2 s JOIN join_mcv_fk2_indexed i ON (s.code = i.code); +ANALYZE join_mcv_fk2; +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_fk2 s + JOIN join_mcv_fk2_indexed i ON (s.id = i.id AND s.code = i.code) + WHERE i.y = 1'); + +DROP TABLE join_mcv_fk2, join_mcv_fk2_indexed; + +-- check semi and anti joins are not corrected, and EXISTS on a primary +-- key, which becomes an inner join, is +CREATE TABLE join_mcv_semi_indexed (id int PRIMARY KEY, b int) + WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_semi (dim_id int) WITH (autovacuum_enabled = off); + +-- b = 1 in a few rows +INSERT INTO join_mcv_semi_indexed +SELECT i, (i <= 10)::int FROM generate_series(1, 100) i; + +-- most rows join the rows where b = 1; the last rows are joined by none +INSERT INTO join_mcv_semi +SELECT CASE WHEN g <= 900 THEN 1 + g % 10 ELSE 11 + g % 50 END +FROM generate_series(1, 1000) g; + +ANALYZE join_mcv_semi_indexed, join_mcv_semi; + +-- Without join statistics +-- An inner join, for comparison. +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_semi f + JOIN join_mcv_semi_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b = 1'); +-- Rows of join_mcv_semi with a match where b = 1, and without one. +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_semi f + WHERE EXISTS (SELECT 1 FROM join_mcv_semi_indexed dim + WHERE dim.id = f.dim_id AND dim.b = 1)'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_semi f + WHERE NOT EXISTS (SELECT 1 FROM join_mcv_semi_indexed dim + WHERE dim.id = f.dim_id AND dim.b = 1)'); +-- Rows of join_mcv_semi_indexed with b = 0 that are joined, and that are not. +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_semi_indexed dim + WHERE dim.b = 0 + AND EXISTS (SELECT 1 FROM join_mcv_semi f + WHERE f.dim_id = dim.id)'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_semi_indexed dim + WHERE dim.b = 0 + AND NOT EXISTS (SELECT 1 FROM join_mcv_semi f + WHERE f.dim_id = dim.id)'); + +-- With join statistics that store every value of b in the join +CREATE STATISTICS join_mcv_semi_stats (mcv) ON dim.b +FROM join_mcv_semi f JOIN join_mcv_semi_indexed dim ON (f.dim_id = dim.id); +ANALYZE join_mcv_semi; +-- An inner join, for comparison. +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_semi f + JOIN join_mcv_semi_indexed dim ON (f.dim_id = dim.id) + WHERE dim.b = 1'); +-- Rows of join_mcv_semi with a match where b = 1: EXISTS on the primary key +-- of join_mcv_semi_indexed becomes an inner join, which the statistics correct +-- as above +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_semi f + WHERE EXISTS (SELECT 1 FROM join_mcv_semi_indexed dim + WHERE dim.id = f.dim_id AND dim.b = 1)'); +-- and without one: an anti join keeps its estimate, as the statistics do not +-- record which rows have no match +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_semi f + WHERE NOT EXISTS (SELECT 1 FROM join_mcv_semi_indexed dim + WHERE dim.id = f.dim_id AND dim.b = 1)'); +-- Rows of join_mcv_semi_indexed with b = 0 that are joined, and that are not: +-- with join_mcv_semi_indexed on the outer side the statistics are not used +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_semi_indexed dim + WHERE dim.b = 0 + AND EXISTS (SELECT 1 FROM join_mcv_semi f + WHERE f.dim_id = dim.id)'); +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_semi_indexed dim + WHERE dim.b = 0 + AND NOT EXISTS (SELECT 1 FROM join_mcv_semi f + WHERE f.dim_id = dim.id)'); + +DROP TABLE join_mcv_semi, join_mcv_semi_indexed; + +-- check join statistics on the partitions are used in a partitionwise join +CREATE TABLE join_mcv_part (a int) PARTITION BY RANGE (a); +CREATE TABLE join_mcv_part_1 PARTITION OF join_mcv_part + FOR VALUES FROM (0) TO (100) WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_part_2 PARTITION OF join_mcv_part + FOR VALUES FROM (100) TO (200) WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_part_indexed (a int, y int) PARTITION BY RANGE (a); +CREATE TABLE join_mcv_part_indexed_1 PARTITION OF join_mcv_part_indexed + FOR VALUES FROM (0) TO (100) WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_part_indexed_2 PARTITION OF join_mcv_part_indexed + FOR VALUES FROM (100) TO (200) WITH (autovacuum_enabled = off); +CREATE INDEX ON join_mcv_part_indexed (a); + +-- y = 1 in a few rows of each partition +INSERT INTO join_mcv_part_indexed +SELECT g, (g % 100 < 10)::int FROM generate_series(0, 199) g; + +-- most rows join the rows where y = 1 +INSERT INTO join_mcv_part +SELECT CASE WHEN g % 10 < 9 THEN g % 10 ELSE g % 100 END + (g % 2) * 100 +FROM generate_series(1, 2000) g; + +ANALYZE join_mcv_part, join_mcv_part_indexed; + +SET enable_partitionwise_join = on; +-- Without join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_part s + JOIN join_mcv_part_indexed i ON (s.a = i.a) + WHERE i.y = 1'); + +-- With join statistics on the partitions +CREATE STATISTICS join_mcv_part_1_stats (mcv) ON i.y +FROM join_mcv_part_1 s JOIN join_mcv_part_indexed_1 i ON (s.a = i.a); +CREATE STATISTICS join_mcv_part_2_stats (mcv) ON i.y +FROM join_mcv_part_2 s JOIN join_mcv_part_indexed_2 i ON (s.a = i.a); +ANALYZE join_mcv_part_1, join_mcv_part_2; +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_part s + JOIN join_mcv_part_indexed i ON (s.a = i.a) + WHERE i.y = 1'); +RESET enable_partitionwise_join; + +DROP TABLE join_mcv_part, join_mcv_part_indexed; + +-- check ANALYZE of the anchor table builds the statistics, from its sample +-- and the matching rows of the probed table, and ANALYZE of the probed +-- table does not +CREATE STATISTICS join_mcv_movies_probe_stats (mcv) ON k.keyword +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword_id = k.id); +ANALYZE join_mcv_keywords; +SELECT d.stxdmcv IS NOT NULL AS has_mcv +FROM pg_statistic_ext s +LEFT JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +WHERE s.stxname = 'join_mcv_movies_probe_stats'; +ANALYZE join_mcv_movies; +SELECT d.stxdmcv IS NOT NULL AS has_mcv +FROM pg_statistic_ext s +LEFT JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +WHERE s.stxname = 'join_mcv_movies_probe_stats'; +DROP STATISTICS join_mcv_movies_probe_stats; + +-- check sampling a join value with many matches looks the value up once +-- through the index on the probed table, however many rows have it +CREATE TABLE join_mcv_dups (k int, a int) WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_dups_indexed (k int, b int) + WITH (autovacuum_enabled = off); +CREATE INDEX ON join_mcv_dups_indexed (k); +-- Four rows with join value 1, one with 2, and one NULL +INSERT INTO join_mcv_dups +VALUES (1, 1), (1, 1), (1, 1), (1, 2), (2, 3), (NULL, 4); +-- many matches each for 1 and 2, and rows with 0 that no row joins +INSERT INTO join_mcv_dups_indexed SELECT 1, 3 FROM generate_series(1, 100); +INSERT INTO join_mcv_dups_indexed SELECT 2, 5 FROM generate_series(1, 100); +INSERT INTO join_mcv_dups_indexed SELECT 0, 1 FROM generate_series(1, 100); +ANALYZE join_mcv_dups, join_mcv_dups_indexed; +CREATE STATISTICS join_mcv_dups_stats (mcv) ON s.a, i.b +FROM join_mcv_dups s JOIN join_mcv_dups_indexed i ON (s.k = i.k); +ANALYZE join_mcv_dups; + +-- one index scan per join value that is not NULL, each match read once +SELECT pg_stat_force_next_flush(); +SELECT idx_scan, idx_tup_read, idx_tup_fetch FROM pg_stat_all_indexes + WHERE indexrelid = 'join_mcv_dups_indexed_k_idx'::regclass; + +-- each row with join value 1 pairs with every match of 1, the row with 2 +-- with the matches of 2, and the NULL row with none +SELECT m.values, m.frequency +FROM pg_statistic_ext s +JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +CROSS JOIN LATERAL pg_mcv_list_items(d.stxdmcv) AS m +WHERE s.stxname = 'join_mcv_dups_stats' +ORDER BY m.frequency DESC, m.values; + +DROP TABLE join_mcv_dups, join_mcv_dups_indexed; + +-- With a hash index on the probed table, a lookup of a join value also +-- returns the matches of every value with the same hash code, and ANALYZE +-- must leave them out. 2775 and 131913 have the same hash code. +SELECT hashint4(2775) = hashint4(131913) AS same_hash_code; +CREATE TABLE join_mcv_hash (k int) WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_hash_indexed (k int) WITH (autovacuum_enabled = off); +CREATE INDEX ON join_mcv_hash_indexed USING hash (k); +INSERT INTO join_mcv_hash VALUES (2775); +INSERT INTO join_mcv_hash_indexed +VALUES (2775), (2775), (2775), (2775), (131913); +ANALYZE join_mcv_hash, join_mcv_hash_indexed; +CREATE STATISTICS join_mcv_hash_stats (mcv) ON i.k +FROM join_mcv_hash s JOIN join_mcv_hash_indexed i ON (s.k = i.k); +ANALYZE join_mcv_hash; + +-- only the matches of 2775 +SELECT m.values, m.frequency +FROM pg_statistic_ext s +JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +CROSS JOIN LATERAL pg_mcv_list_items(d.stxdmcv) AS m +WHERE s.stxname = 'join_mcv_hash_stats'; + +DROP TABLE join_mcv_hash, join_mcv_hash_indexed; + +-- check a join larger than the sample: with statistics target 10, ANALYZE +-- samples 1000 of the 100000 joined rows, so the frequencies vary from run +-- to run +CREATE TABLE join_mcv_big (k int) WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_big_indexed (k int, b int) + WITH (autovacuum_enabled = off); +CREATE INDEX ON join_mcv_big_indexed (k); +-- one join value, whose matches have b in 1..4 in equal shares +INSERT INTO join_mcv_big SELECT 1 FROM generate_series(1, 40); +INSERT INTO join_mcv_big_indexed +SELECT 1, (g - 1) / 625 + 1 FROM generate_series(1, 2500) g; +ANALYZE join_mcv_big, join_mcv_big_indexed; +CREATE STATISTICS join_mcv_big_stats (mcv) ON i.b +FROM join_mcv_big s JOIN join_mcv_big_indexed i ON (s.k = i.k); +ALTER STATISTICS join_mcv_big_stats SET STATISTICS 10; +ANALYZE join_mcv_big; + +-- the sample has every value of b, not only the first ones, and exactly the +-- rows the target allows, so every frequency is a multiple of 1/1000 +SELECT count(*) AS b_values, + bool_and(abs(m.frequency * 1000 - round(m.frequency * 1000)) < 1e-6) + AS in_thousandths +FROM pg_statistic_ext s +JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +CROSS JOIN LATERAL pg_mcv_list_items(d.stxdmcv) AS m +WHERE s.stxname = 'join_mcv_big_stats'; + +DROP TABLE join_mcv_big, join_mcv_big_indexed; + +-- check the join column of the probed table needs an index ANALYZE can look +-- the join values up in: without one, CREATE warns and ANALYZE skips the +-- object +-- XXX Sampling without an index, by scanning the probed table, is planned; +-- the cases skipped here will then build. +CREATE TABLE join_mcv_noidx (id INTEGER, val TEXT); +-- a few rows, joined by the movies with keyword_id 1 +INSERT INTO join_mcv_noidx SELECT 1, 'x' FROM generate_series(1, 20); +CREATE STATISTICS join_mcv_noidx_stats (mcv) ON n.val +FROM join_mcv_movies m JOIN join_mcv_noidx n ON (m.keyword_id = n.id); +ANALYZE join_mcv_movies; +SELECT d.stxdmcv IS NOT NULL AS has_mcv +FROM pg_statistic_ext s +LEFT JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +WHERE s.stxname = 'join_mcv_noidx_stats'; +-- None of these is suitable: a BRIN index does not support equality lookup, +-- a partial index does not cover every row, and a multi-column index can be +-- looked up only by its leading column. +CREATE INDEX join_mcv_noidx_brin ON join_mcv_noidx USING brin (id); +CREATE INDEX join_mcv_noidx_partial ON join_mcv_noidx (id) WHERE id > 0; +CREATE INDEX join_mcv_noidx_nonlead ON join_mcv_noidx (val, id); +ANALYZE join_mcv_movies; +SELECT d.stxdmcv IS NOT NULL AS has_mcv +FROM pg_statistic_ext s +LEFT JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +WHERE s.stxname = 'join_mcv_noidx_stats'; +-- A btree index with the join column leading is suitable. +CREATE INDEX join_mcv_noidx_wide ON join_mcv_noidx (id, val); +ANALYZE join_mcv_movies; +SELECT d.stxdmcv IS NOT NULL AS has_mcv +FROM pg_statistic_ext s +LEFT JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +WHERE s.stxname = 'join_mcv_noidx_stats'; +DROP INDEX join_mcv_noidx_wide; +-- With the last usable index gone, ANALYZE warns and keeps the data it has +ANALYZE join_mcv_movies; +SELECT d.stxdmcv IS NOT NULL AS has_mcv +FROM pg_statistic_ext s +LEFT JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +WHERE s.stxname = 'join_mcv_noidx_stats'; +DROP TABLE join_mcv_noidx; + +-- check ALTER TABLE ... ALTER COLUMN TYPE re-creates the statistics objects +-- that use the column, as for single-table statistics, and leaves the +-- others alone +CREATE STATISTICS join_mcv_movies_alter_stats (mcv) ON mk.year +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword_id = k.id); +ANALYZE join_mcv_movies; + +ALTER TABLE join_mcv_keywords ALTER COLUMN phonetic_code TYPE varchar(10); + +SELECT d.stxdmcv IS NOT NULL AS has_mcv +FROM pg_statistic_ext s +LEFT JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +WHERE s.stxname = 'join_mcv_movies_alter_stats'; + +-- A column of the statistics: re-created without data +ALTER TABLE join_mcv_movies ALTER COLUMN year TYPE numeric; + +SELECT d.stxdmcv IS NOT NULL AS has_mcv +FROM pg_statistic_ext s +LEFT JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +WHERE s.stxname = 'join_mcv_movies_alter_stats'; + +DROP STATISTICS join_mcv_movies_alter_stats; + +-- Changing the type of a column of the probed table re-creates the +-- statistics, as for a column of the table they belong to +CREATE STATISTICS join_mcv_movies_alter_dim_stats (mcv) ON k.keyword +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword_id = k.id); +ANALYZE join_mcv_movies; +SELECT pg_get_statisticsobjdef(s.oid, false), d.stxdmcv IS NOT NULL AS has_mcv +FROM pg_statistic_ext s LEFT JOIN pg_statistic_ext_data d ON d.stxoid = s.oid +WHERE s.stxname = 'join_mcv_movies_alter_dim_stats'; +BEGIN; +ALTER TABLE join_mcv_keywords ALTER COLUMN keyword TYPE varchar(20); +-- ALTER TABLE does not lock join_mcv_movies exclusively: it is locked only +-- as for CREATE STATISTICS +SELECT mode FROM pg_locks +WHERE pid = pg_backend_pid() AND relation = 'join_mcv_movies'::regclass +ORDER BY mode; +COMMIT; +SELECT pg_get_statisticsobjdef(s.oid, false), d.stxdmcv IS NOT NULL AS has_mcv +FROM pg_statistic_ext s LEFT JOIN pg_statistic_ext_data d ON d.stxoid = s.oid +WHERE s.stxname = 'join_mcv_movies_alter_dim_stats'; +DROP STATISTICS join_mcv_movies_alter_dim_stats; + +-- A self-join's statistics are re-created once, although both sides change +CREATE STATISTICS join_mcv_movies_alter_self_stats (mcv) ON b.keyword +FROM join_mcv_keywords a JOIN join_mcv_keywords b ON (a.id = b.id); +ALTER TABLE join_mcv_keywords ALTER COLUMN id TYPE bigint; +SELECT pg_get_statisticsobjdef(oid, false) FROM pg_statistic_ext +WHERE stxname = 'join_mcv_movies_alter_self_stats'; +DROP STATISTICS join_mcv_movies_alter_self_stats; + +-- A join column may change to a type that can still be compared with the +-- other one (join_mcv_keywords.id is bigint since the case above); otherwise +-- ALTER TABLE fails, as the statistics cannot be re-created +CREATE STATISTICS join_mcv_movies_alter_key_stats (mcv) ON k.keyword +FROM join_mcv_movies mk JOIN join_mcv_keywords k ON (mk.keyword_id = k.id); +ALTER TABLE join_mcv_movies ALTER COLUMN keyword_id TYPE bigint; +ALTER TABLE join_mcv_movies ALTER COLUMN keyword_id TYPE numeric; +DROP STATISTICS join_mcv_movies_alter_key_stats; + +DROP TABLE join_mcv_movies, join_mcv_keywords; + +-- check tables with inheritance children are refused like partitioned +-- tables, whichever table has them, and children added later make ANALYZE +-- skip the statistics +CREATE TABLE join_mcv_inh (dim_id int); +CREATE TABLE join_mcv_inh_indexed (id int PRIMARY KEY, label text); +INSERT INTO join_mcv_inh_indexed +SELECT g, 'label ' || g % 3 FROM generate_series(1, 99) g; +INSERT INTO join_mcv_inh SELECT 1 + g % 99 FROM generate_series(1, 1000) g; +CREATE TABLE join_mcv_inh_child () INHERITS (join_mcv_inh); +CREATE STATISTICS join_mcv_inh_anchor_stats (mcv) ON d.label +FROM join_mcv_inh f JOIN join_mcv_inh_indexed d ON (f.dim_id = d.id); +DROP TABLE join_mcv_inh_child; +CREATE TABLE join_mcv_inh_indexed_child () INHERITS (join_mcv_inh_indexed); +CREATE STATISTICS join_mcv_inh_probed_stats (mcv) ON d.label +FROM join_mcv_inh f JOIN join_mcv_inh_indexed d ON (f.dim_id = d.id); +DROP TABLE join_mcv_inh_indexed_child; + +-- Children added later: the statistics are skipped, rather than built +-- without the children's rows +CREATE STATISTICS join_mcv_inh_later_stats (mcv) ON d.label +FROM join_mcv_inh f JOIN join_mcv_inh_indexed d ON (f.dim_id = d.id); +CREATE TABLE join_mcv_inh_indexed_child () INHERITS (join_mcv_inh_indexed); +INSERT INTO join_mcv_inh_indexed_child +SELECT g, 'child' FROM generate_series(1, 99) g; +ANALYZE join_mcv_inh; +SELECT most_common_vals FROM pg_stats_ext +WHERE statistics_name = 'join_mcv_inh_later_stats'; +-- Children of the anchor table are reported once, although ANALYZE goes over +-- that table both with and without its children +CREATE TABLE join_mcv_inh_child () INHERITS (join_mcv_inh); +ANALYZE join_mcv_inh; + +DROP TABLE join_mcv_inh, join_mcv_inh_indexed CASCADE; + +-- Permission tests. CREATE requires ownership of the anchor table and +-- SELECT on the probed table. ANALYZE refreshes the statistics only if the +-- owner still has SELECT on the probed table; otherwise it warns and skips +-- them, leaving the existing data in place. +-- +CREATE ROLE regress_join_mcv_user; +CREATE ROLE regress_join_mcv_user2; + +CREATE TABLE join_mcv_perm (id int, val int) WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_perm_indexed (id int PRIMARY KEY, label text) + WITH (autovacuum_enabled = off); +INSERT INTO join_mcv_perm SELECT i, i % 10 FROM generate_series(1, 1000) i; +INSERT INTO join_mcv_perm_indexed SELECT i, 'l' FROM generate_series(1, 1000) i; +ANALYZE join_mcv_perm, join_mcv_perm_indexed; + +GRANT CREATE ON SCHEMA public TO regress_join_mcv_user, regress_join_mcv_user2; +GRANT SELECT ON join_mcv_perm, join_mcv_perm_indexed TO regress_join_mcv_user; + +-- Not the anchor owner: CREATE fails +SET SESSION AUTHORIZATION regress_join_mcv_user; +CREATE STATISTICS join_mcv_perm_stats (mcv) ON a.val +FROM join_mcv_perm a JOIN join_mcv_perm_indexed i ON (a.id = i.id); +RESET SESSION AUTHORIZATION; +ALTER TABLE join_mcv_perm OWNER TO regress_join_mcv_user; + +-- Anchor owner, no SELECT on the probed table: CREATE fails +REVOKE SELECT ON join_mcv_perm_indexed FROM regress_join_mcv_user; +SET SESSION AUTHORIZATION regress_join_mcv_user; +CREATE STATISTICS join_mcv_perm_stats (mcv) ON a.val +FROM join_mcv_perm a JOIN join_mcv_perm_indexed i ON (a.id = i.id); +RESET SESSION AUTHORIZATION; + +-- Anchor owner, SELECT on the probed table: CREATE succeeds +GRANT SELECT ON join_mcv_perm_indexed TO regress_join_mcv_user; +SET SESSION AUTHORIZATION regress_join_mcv_user; +CREATE STATISTICS join_mcv_perm_stats (mcv) ON a.val +FROM join_mcv_perm a JOIN join_mcv_perm_indexed i ON (a.id = i.id); +RESET SESSION AUTHORIZATION; + +-- Owner has SELECT: ANALYZE builds the statistics +ANALYZE join_mcv_perm; +SELECT d.stxdmcv IS NOT NULL AS has_mcv +FROM pg_statistic_ext s +LEFT JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +WHERE s.stxname = 'join_mcv_perm_stats'; + +-- Anchor owner changed: ANALYZE still refreshes (no warning), and ALTER +-- COLUMN TYPE still re-creates the object for its owner, without data until +-- the next ANALYZE +ALTER TABLE join_mcv_perm OWNER TO CURRENT_USER; +ANALYZE join_mcv_perm; +ALTER TABLE join_mcv_perm ALTER COLUMN val TYPE bigint; +SELECT stxname, stxowner::regrole FROM pg_statistic_ext +WHERE stxname = 'join_mcv_perm_stats'; +ANALYZE join_mcv_perm; + +-- Statistics owner changed to a role without SELECT on the probed table: +-- ANALYZE skips +ALTER STATISTICS join_mcv_perm_stats OWNER TO regress_join_mcv_user2; +ANALYZE join_mcv_perm; +SELECT d.stxdmcv IS NOT NULL AS has_mcv +FROM pg_statistic_ext s +LEFT JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +WHERE s.stxname = 'join_mcv_perm_stats'; + +-- Owner granted SELECT on the probed table: ANALYZE refreshes (no warning) +GRANT SELECT ON join_mcv_perm_indexed TO regress_join_mcv_user2; +ANALYZE join_mcv_perm; + +-- Owner's SELECT on the probed table revoked: ANALYZE skips, while ALTER +-- COLUMN TYPE still re-creates the object (no error) +REVOKE SELECT ON join_mcv_perm_indexed FROM regress_join_mcv_user2; +ANALYZE join_mcv_perm; +SELECT d.stxdmcv IS NOT NULL AS has_mcv +FROM pg_statistic_ext s +LEFT JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +WHERE s.stxname = 'join_mcv_perm_stats'; +ALTER TABLE join_mcv_perm ALTER COLUMN val TYPE int; + +DROP STATISTICS join_mcv_perm_stats; +DROP TABLE join_mcv_perm, join_mcv_perm_indexed; +REVOKE CREATE ON SCHEMA public + FROM regress_join_mcv_user, regress_join_mcv_user2; +DROP ROLE regress_join_mcv_user, regress_join_mcv_user2; + +-- check the owner of a joined table can drop their index, Tomas Vondra's +-- scenario: owner1 owns the anchor and the statistics object, owner2 the +-- other table and its index +CREATE ROLE regress_join_mcv_owner1; +CREATE ROLE regress_join_mcv_owner2; +GRANT USAGE, CREATE ON SCHEMA public TO regress_join_mcv_owner1; +GRANT USAGE, CREATE ON SCHEMA public TO regress_join_mcv_owner2; +SET ROLE regress_join_mcv_owner2; +CREATE TABLE join_mcv_owners_indexed (a int, b int); +INSERT INTO join_mcv_owners_indexed +SELECT g, g % 2 FROM generate_series(1, 100) g; +CREATE INDEX join_mcv_owners_idx ON join_mcv_owners_indexed (a); +RESET ROLE; +GRANT SELECT ON join_mcv_owners_indexed TO regress_join_mcv_owner1; +SET ROLE regress_join_mcv_owner1; +CREATE TABLE join_mcv_owners (a int, b int); +INSERT INTO join_mcv_owners +SELECT g % 100 + 1, g % 3 FROM generate_series(1, 1000) g; +CREATE STATISTICS join_mcv_owners_stats (mcv) ON s.a, s.b, i.a, i.b + FROM join_mcv_owners s JOIN join_mcv_owners_indexed i ON (s.a = i.a); +ANALYZE join_mcv_owners; +-- one dependency row per column, although s.a and i.a are both join and +-- statistics columns +SELECT pg_describe_object(refclassid, refobjid, refobjsubid) AS ref +FROM pg_depend +WHERE objid = (SELECT oid FROM pg_statistic_ext + WHERE stxname = 'join_mcv_owners_stats') + AND refobjsubid <> 0 +ORDER BY 1; +SET ROLE regress_join_mcv_owner2; +DROP INDEX join_mcv_owners_idx; +-- The statistics object stays, and is skipped until owner2 has an index +-- again +SET ROLE regress_join_mcv_owner1; +ANALYZE join_mcv_owners; +SELECT stxname FROM pg_statistic_ext WHERE stxname = 'join_mcv_owners_stats'; +SET ROLE regress_join_mcv_owner2; +CREATE INDEX join_mcv_owners_idx ON join_mcv_owners_indexed (a); +SET ROLE regress_join_mcv_owner1; +ANALYZE join_mcv_owners; +RESET ROLE; +SELECT d.stxdmcv IS NOT NULL AS has_mcv +FROM pg_statistic_ext s +LEFT JOIN pg_statistic_ext_data d ON (s.oid = d.stxoid) +WHERE s.stxname = 'join_mcv_owners_stats'; +DROP TABLE join_mcv_owners, join_mcv_owners_indexed; +REVOKE USAGE, CREATE ON SCHEMA public + FROM regress_join_mcv_owner1, regress_join_mcv_owner2; +DROP ROLE regress_join_mcv_owner1, regress_join_mcv_owner2; + +-- check join statistics do not reveal rows that row-level security hides +CREATE FUNCTION op_leak(int, int) RETURNS bool + AS 'BEGIN RAISE NOTICE ''op_leak => %, %'', $1, $2; RETURN $1 < $2; END' + LANGUAGE plpgsql; +-- The join MCV estimator only considers operators that have a restriction +-- estimator and belong to a btree operator family. +CREATE OPERATOR <<< (procedure = op_leak, leftarg = int, rightarg = int, + restrict = scalarltsel); +CREATE OPERATOR FAMILY leak_ops USING btree; +ALTER OPERATOR FAMILY leak_ops USING btree ADD OPERATOR 1 <<< (int4, int4); + +CREATE ROLE regress_join_mcv_rls_owner; +CREATE ROLE regress_join_mcv_rls_anchor_owner; +CREATE SCHEMA join_rls AUTHORIZATION regress_join_mcv_rls_owner; +SET ROLE regress_join_mcv_rls_owner; +CREATE TABLE join_rls.anchor (id int, filler int); +CREATE TABLE join_rls.secret (id int, secret text); +INSERT INTO join_rls.anchor SELECT g, g FROM generate_series(1, 100) g; +INSERT INTO join_rls.secret + SELECT g, CASE WHEN g <= 50 THEN 'secret-1' ELSE 'secret-2' END + FROM generate_series(1, 100) g; +CREATE INDEX ON join_rls.secret (id); +-- No column statistics on filler, so op_leak can only see the join MCV's +-- values +ALTER TABLE join_rls.anchor ALTER COLUMN filler SET STATISTICS 0; +CREATE STATISTICS join_rls.secret_stats (mcv) ON anchor.filler, secret.secret + FROM join_rls.anchor JOIN join_rls.secret ON anchor.id = secret.id; +ANALYZE join_rls.anchor; + +-- The owner can read secret, so it sees the statistics. +SELECT statistics_name, most_common_vals IS NOT NULL AS has_mcv +FROM pg_stats_ext WHERE statistics_schemaname = 'join_rls'; + +-- Owning only anchor is not enough to see them +RESET ROLE; +ALTER TABLE join_rls.anchor OWNER TO regress_join_mcv_rls_anchor_owner; +SET ROLE regress_join_mcv_rls_anchor_owner; +SELECT statistics_name, most_common_vals IS NOT NULL AS has_mcv +FROM pg_stats_ext WHERE statistics_schemaname = 'join_rls'; +RESET ROLE; +ALTER TABLE join_rls.anchor OWNER TO regress_join_mcv_rls_owner; +SET ROLE regress_join_mcv_rls_owner; + +-- Now nobody can read any rows of secret, not even its owner. (With no +-- policy at all, the planner would prove the join empty and never estimate +-- it.) +ALTER TABLE join_rls.secret ENABLE ROW LEVEL SECURITY; +ALTER TABLE join_rls.secret FORCE ROW LEVEL SECURITY; +CREATE POLICY nobody ON join_rls.secret USING (id < 0); +SELECT count(*) AS visible_secret_rows FROM join_rls.secret; + +-- So its values must not show up in pg_stats_ext, +SELECT statistics_name, most_common_vals IS NOT NULL AS has_mcv +FROM pg_stats_ext WHERE statistics_schemaname = 'join_rls'; + +-- nor be passed to a leaky operator while planning: no op_leak NOTICE. +EXPLAIN (COSTS OFF) +SELECT * FROM join_rls.anchor JOIN join_rls.secret ON anchor.id = secret.id + WHERE anchor.filler <<< 5; +RESET ROLE; + +DROP SCHEMA join_rls CASCADE; +DROP ROLE regress_join_mcv_rls_owner, regress_join_mcv_rls_anchor_owner; +DROP OPERATOR FAMILY leak_ops USING btree; +DROP OPERATOR <<< (int4, int4); +DROP FUNCTION op_leak(int, int); + +-- These objects are not dropped, so that the pg_upgrade test dumps and +-- restores join statistics of each of these shapes; the same holds for +-- join_mcv_stored and join_mcv_filters above. +-- +-- check a cross-type join, a float4 anchor column joined to a float8 column +-- through the cross-type equality operator +CREATE TABLE join_mcv_crosstype_indexed (id float8 PRIMARY KEY, + color text NOT NULL) + WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_crosstype (id serial, dim_id float4 NOT NULL) + WITH (autovacuum_enabled = off); + +-- three colors +INSERT INTO join_mcv_crosstype_indexed +SELECT i::float8, CASE WHEN i <= 5 THEN 'red' + WHEN i <= 10 THEN 'blue' + ELSE 'green' END +FROM generate_series(1, 15) i; + +-- most rows reference the red rows +INSERT INTO join_mcv_crosstype (dim_id) +SELECT CASE WHEN i % 5 < 4 THEN (i % 5 + 1)::float4 + ELSE (6 + i % 10)::float4 END +FROM generate_series(1, 500) i; + +ANALYZE join_mcv_crosstype_indexed; +ANALYZE join_mcv_crosstype; + +-- Without join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_crosstype f, join_mcv_crosstype_indexed d + WHERE f.dim_id = d.id AND d.color = ''red'' +'); + +CREATE STATISTICS join_mcv_crosstype_stats (mcv) ON f.dim_id, d.color +FROM join_mcv_crosstype f + JOIN join_mcv_crosstype_indexed d ON (f.dim_id = d.id); + +ANALYZE join_mcv_crosstype; + +-- With join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_crosstype f, join_mcv_crosstype_indexed d + WHERE f.dim_id = d.id AND d.color = ''red'' +'); + +-- The join condition may name the joined table first: the statistics are the +-- same +CREATE STATISTICS join_mcv_crosstype_swapped_stats (mcv) ON f.dim_id, d.color +FROM join_mcv_crosstype f + JOIN join_mcv_crosstype_indexed d ON (d.id = f.dim_id); +ANALYZE join_mcv_crosstype; +SELECT a.most_common_vals = b.most_common_vals AS same_vals, + a.most_common_freqs = b.most_common_freqs AS same_freqs +FROM pg_stats_ext a, pg_stats_ext b +WHERE a.statistics_name = 'join_mcv_crosstype_stats' + AND b.statistics_name = 'join_mcv_crosstype_swapped_stats'; +DROP STATISTICS join_mcv_crosstype_swapped_stats; + +-- Leave join_mcv_crosstype, join_mcv_crosstype_indexed and +-- join_mcv_crosstype_stats in place for pg_upgrade testing. + +-- check a join of binary-compatible types, a varchar column joined to a +-- text key: the parser relabels the varchar column to text, as the planner +-- does for the same join in a query +CREATE TABLE join_mcv_relabel_indexed (code text PRIMARY KEY, + color text NOT NULL) + WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_relabel (id serial, dim_code varchar(10) NOT NULL) + WITH (autovacuum_enabled = off); + +-- three colors +INSERT INTO join_mcv_relabel_indexed +SELECT 'c' || i, CASE WHEN i <= 5 THEN 'red' + WHEN i <= 10 THEN 'blue' + ELSE 'green' END +FROM generate_series(1, 15) i; + +-- most rows reference the red rows +INSERT INTO join_mcv_relabel (dim_code) +SELECT 'c' || CASE WHEN i % 5 < 4 THEN i % 5 + 1 ELSE 6 + i % 10 END +FROM generate_series(1, 500) i; + +ANALYZE join_mcv_relabel_indexed; +ANALYZE join_mcv_relabel; + +-- Without join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_relabel f, join_mcv_relabel_indexed d + WHERE f.dim_code = d.code AND d.color = ''red'' +'); + +CREATE STATISTICS join_mcv_relabel_stats (mcv) ON d.color +FROM join_mcv_relabel f + JOIN join_mcv_relabel_indexed d ON (f.dim_code = d.code); +SELECT pg_get_statisticsobjdef(oid, false) FROM pg_statistic_ext +WHERE stxname = 'join_mcv_relabel_stats'; +-- Dependencies are recorded on both join columns, the varchar one included +SELECT pg_describe_object(refclassid, refobjid, refobjsubid) AS ref +FROM pg_depend +WHERE objid = (SELECT oid FROM pg_statistic_ext + WHERE stxname = 'join_mcv_relabel_stats') + AND refobjsubid <> 0 +ORDER BY 1; +-- The join condition is stored with its collation +SELECT substring(stxjoinconds from ':inputcollid \d+') AS inputcollid +FROM pg_statistic_ext WHERE stxname = 'join_mcv_relabel_stats'; +-- A column merged by USING can be used by a later join: it stands for the +-- varchar column +CREATE TABLE join_mcv_relabel_code (code varchar(10)); +CREATE STATISTICS join_mcv_relabel_using_stats (mcv) ON d2.color +FROM join_mcv_relabel_code JOIN join_mcv_relabel_indexed USING (code) + JOIN join_mcv_relabel_indexed d2 USING (code); +SELECT pg_get_statisticsobjdef(oid, false) FROM pg_statistic_ext +WHERE stxname = 'join_mcv_relabel_using_stats'; +DROP TABLE join_mcv_relabel_code; + +ANALYZE join_mcv_relabel; + +-- With join statistics +SELECT * FROM check_estimated_rows(' + SELECT * FROM join_mcv_relabel f, join_mcv_relabel_indexed d + WHERE f.dim_code = d.code AND d.color = ''red'' +'); + +-- LIKE INCLUDING STATISTICS copies only the single-table statistics: join +-- statistics describe a join, not the table +CREATE STATISTICS join_mcv_relabel_single_stats ON id, dim_code +FROM join_mcv_relabel; +CREATE TABLE join_mcv_relabel_copy + (LIKE join_mcv_relabel INCLUDING STATISTICS); +SELECT stxname, pg_get_statisticsobjdef(oid, false) FROM pg_statistic_ext +WHERE stxrelid = 'join_mcv_relabel_copy'::regclass; +DROP TABLE join_mcv_relabel_copy; +DROP STATISTICS join_mcv_relabel_single_stats; + +-- Leave join_mcv_relabel, join_mcv_relabel_indexed and join_mcv_relabel_stats +-- in place for pg_upgrade testing. + +-- check same-named tables in different schemas get distinct aliases in the +-- deparsed definition + +CREATE SCHEMA join_mcv_schema1; +CREATE SCHEMA join_mcv_schema2; + +CREATE TABLE join_mcv_schema1.t (a int PRIMARY KEY) + WITH (autovacuum_enabled = off); +CREATE TABLE join_mcv_schema2.t (b int) + WITH (autovacuum_enabled = off); +CREATE INDEX ON join_mcv_schema2.t (b); + +INSERT INTO join_mcv_schema1.t SELECT generate_series(1, 100); +INSERT INTO join_mcv_schema2.t SELECT generate_series(1, 100); + +ANALYZE join_mcv_schema1.t; +ANALYZE join_mcv_schema2.t; + +CREATE STATISTICS join_mcv_schemas_stats (mcv) ON join_mcv_schema2.t.b +FROM join_mcv_schema1.t + JOIN join_mcv_schema2.t ON (join_mcv_schema1.t.a = join_mcv_schema2.t.b); + +SELECT pg_get_statisticsobjdef(oid) FROM pg_statistic_ext +WHERE stxname = 'join_mcv_schemas_stats'; + +-- Leave the join_mcv_schema1 and join_mcv_schema2 schemas and +-- join_mcv_schemas_stats in place for pg_upgrade testing. + +-- check the auto-generated name of a join statistics object qualifies +-- columns with their relation, written before the first column and again +-- whenever the relation changes (single-table naming is covered in +-- stats_ext) +CREATE TABLE join_mcv_auto1 (a INTEGER, b INTEGER, c INTEGER); +CREATE TABLE join_mcv_auto2 (d INTEGER, e INTEGER, f INTEGER); +CREATE INDEX ON join_mcv_auto1 (a); +CREATE INDEX ON join_mcv_auto2 (d); + +-- Two columns from each relation: each relation name is written once. +CREATE STATISTICS (mcv) ON a1.b, a1.c, a2.e, a2.f + FROM join_mcv_auto1 a1 JOIN join_mcv_auto2 a2 ON (a1.a = a2.d); + +-- Columns alternating between the relations: the relation name is written +-- again each time it changes. +CREATE STATISTICS (mcv) ON a2.e, a1.b, a2.f + FROM join_mcv_auto1 a1 JOIN join_mcv_auto2 a2 ON (a1.a = a2.d); + +-- Columns from only one of the relations: that relation name is written once. +CREATE STATISTICS (mcv) ON a2.e, a2.f + FROM join_mcv_auto1 a1 JOIN join_mcv_auto2 a2 ON (a1.a = a2.d); + +\dX join_mcv_auto* + +-- Leave join_mcv_auto1, join_mcv_auto2 and their statistics in place for +-- pg_upgrade testing. + +-- check definitions printed differently than written: tables in another +-- order, and several join conditions +CREATE TABLE join_mcv_shape (id INTEGER PRIMARY KEY, dim1_id INTEGER); +CREATE TABLE join_mcv_shape_dim1 (id INTEGER PRIMARY KEY, val TEXT); +CREATE TABLE join_mcv_shape_dim2 (id INTEGER PRIMARY KEY, label TEXT); +CREATE STATISTICS join_mcv_shape_nested_stats (mcv) ON d1.val, d2.label + FROM join_mcv_shape f + JOIN (join_mcv_shape_dim1 d1 JOIN join_mcv_shape_dim2 d2 ON (d1.id = d2.id)) + ON (f.dim1_id = d2.id); +CREATE STATISTICS join_mcv_shape_multiclause_stats (mcv) ON d1.val + FROM join_mcv_shape f + JOIN join_mcv_shape_dim1 d1 ON (f.dim1_id = d1.id AND f.id = d1.id); +SELECT stxname, stxjoinrels::regclass[], pg_get_statisticsobjdef(oid, false) +FROM pg_statistic_ext WHERE stxname LIKE 'join_mcv_shape%' ORDER BY stxname; +-- Leave join_mcv_shape, join_mcv_shape_dim1, join_mcv_shape_dim2 and their +-- statistics in place for pg_upgrade testing. diff --git a/src/test/regress/sql/stats_import.sql b/src/test/regress/sql/stats_import.sql index 2f0c5885fe5..7d601ab0953 100644 --- a/src/test/regress/sql/stats_import.sql +++ b/src/test/regress/sql/stats_import.sql @@ -3287,4 +3287,122 @@ SELECT expr, n_distinct FROM pg_stats_ext_exprs WHERE statistics_schemaname = 'stats_import' AND statistics_name = 'test_fold_stat' ORDER BY expr; +-- Join statistics: copy their statistics to clones and compare, like for +-- test_stat above. The columns of join statistics can come from the joined +-- table rather than from the table the statistics belong to. +CREATE TABLE stats_import.tail_dim ( + id INTEGER PRIMARY KEY, + label TEXT NOT NULL +) WITH (autovacuum_enabled = false); +CREATE TABLE stats_import.tail_fact ( + dim_id INTEGER NOT NULL +) WITH (autovacuum_enabled = false); +INSERT INTO stats_import.tail_dim VALUES + (1, 'common_a'), (2, 'common_b'), + (3, 'other_a'), (4, 'other_b'), + (5, 'extra_1'), (6, 'extra_2'); +INSERT INTO stats_import.tail_fact (dim_id) +SELECT 1 FROM generate_series(1, 90) +UNION ALL SELECT 2 FROM generate_series(1, 70) +UNION ALL SELECT 3 FROM generate_series(1, 25) +UNION ALL SELECT 4 FROM generate_series(1, 15); +ANALYZE stats_import.tail_dim; +ANALYZE stats_import.tail_fact; +-- tail_dim.label is column 2, and tail_fact has only one column +CREATE STATISTICS stats_import.jstats_tail (mcv) ON d.label +FROM stats_import.tail_fact f JOIN stats_import.tail_dim d ON (f.dim_id = d.id); +ALTER STATISTICS stats_import.jstats_tail SET STATISTICS 2; +ANALYZE stats_import.tail_fact; + +CREATE TABLE stats_import.test_orders (id int, test_id int, qty int) + WITH (autovacuum_enabled = false); +INSERT INTO stats_import.test_orders + SELECT g, g % 4 + 1, g % 2 FROM generate_series(1, 80) g; +-- Columns from both tables; test.name is column 2, which in test_orders is +-- the integer test_id +CREATE STATISTICS stats_import.test_orders_stat (mcv) ON o.qty, t.name +FROM stats_import.test_orders o JOIN stats_import.test t ON (o.test_id = t.id); +ANALYZE stats_import.test_orders; + +CREATE TABLE stats_import.tail_fact_clone (LIKE stats_import.tail_fact) + WITH (autovacuum_enabled = false); +CREATE TABLE stats_import.tail_dim_clone (LIKE stats_import.tail_dim) + WITH (autovacuum_enabled = false); +CREATE TABLE stats_import.test_orders_clone (LIKE stats_import.test_orders) + WITH (autovacuum_enabled = false); +-- Join statistics need an index on the joined table's join column +CREATE INDEX ON stats_import.tail_dim_clone (id); +CREATE INDEX ON stats_import.test_clone (id); +CREATE STATISTICS stats_import.jstats_tail_clone (mcv) ON d.label +FROM stats_import.tail_fact_clone f JOIN stats_import.tail_dim_clone d + ON (f.dim_id = d.id); +CREATE STATISTICS stats_import.test_orders_stat_clone (mcv) ON o.qty, t.name +FROM stats_import.test_orders_clone o JOIN stats_import.test_clone t + ON (o.test_id = t.id); + +-- The statistics to copy +SELECT statistics_name, most_common_vals, most_common_freqs FROM pg_stats_ext +WHERE statistics_schemaname = 'stats_import' + AND statistics_name IN ('jstats_tail', 'test_orders_stat') +ORDER BY statistics_name; + +SELECT pg_catalog.pg_restore_extended_stats( + 'schemaname', 'stats_import', + 'relname', 'tail_fact_clone', + 'statistics_schemaname', 'stats_import', + 'statistics_name', 'jstats_tail_clone', + 'inherited', e.inherited, + 'most_common_vals', e.most_common_vals, + 'most_common_freqs', e.most_common_freqs, + 'most_common_base_freqs', e.most_common_base_freqs) +FROM pg_stats_ext AS e +WHERE e.statistics_schemaname = 'stats_import' + AND e.statistics_name = 'jstats_tail'; +SELECT pg_catalog.pg_restore_extended_stats( + 'schemaname', 'stats_import', + 'relname', 'test_orders_clone', + 'statistics_schemaname', 'stats_import', + 'statistics_name', 'test_orders_stat_clone', + 'inherited', e.inherited, + 'most_common_vals', e.most_common_vals, + 'most_common_freqs', e.most_common_freqs, + 'most_common_base_freqs', e.most_common_base_freqs) +FROM pg_stats_ext AS e +WHERE e.statistics_schemaname = 'stats_import' + AND e.statistics_name = 'test_orders_stat'; + +-- The clones have the same statistics: no differences +SELECT statname, (stats).* +FROM stats_import.pg_stats_ext_get_difference('jstats_tail', 'jstats_tail_clone') +\gx +SELECT statname, (stats).* +FROM stats_import.pg_stats_ext_get_difference('test_orders_stat', + 'test_orders_stat_clone') +\gx + +-- Restoring join statistics needs MAINTAIN on the anchor table only: it +-- writes the given values and reads nothing from the other tables +CREATE ROLE regress_test_join_restore; +GRANT USAGE ON SCHEMA stats_import TO regress_test_join_restore; +GRANT MAINTAIN ON stats_import.tail_fact_clone TO regress_test_join_restore; +SET ROLE regress_test_join_restore; +SELECT has_table_privilege('stats_import.tail_dim_clone', 'SELECT, MAINTAIN') + AS dim_privileges; +SELECT pg_catalog.pg_restore_extended_stats( + 'schemaname', 'stats_import', + 'relname', 'tail_fact_clone', + 'statistics_schemaname', 'stats_import', + 'statistics_name', 'jstats_tail_clone', + 'inherited', false, + 'most_common_vals', '{{common_a}}'::text[], + 'most_common_freqs', '{1}'::double precision[], + 'most_common_base_freqs', '{1}'::double precision[]); +RESET ROLE; +SELECT most_common_vals FROM pg_stats_ext +WHERE statistics_schemaname = 'stats_import' + AND statistics_name = 'jstats_tail_clone'; +REVOKE MAINTAIN ON stats_import.tail_fact_clone FROM regress_test_join_restore; +REVOKE USAGE ON SCHEMA stats_import FROM regress_test_join_restore; +DROP ROLE regress_test_join_restore; + DROP SCHEMA stats_import CASCADE; diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index f10eb469266..08f3062cef8 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -1404,11 +1404,17 @@ JoinCostWorkspace JoinDomain JoinExpr JoinHashEntry +JoinMCVFilter JoinPath JoinPathExtraData +JoinSampleColumn +JoinStatRel JoinState JoinTreeItem JoinType +JoinValueGroup +JoinValueTable_hash +JoinValueTable_iterator JsObject JsValue JsonAggConstructor -- 2.54.0 (Apple Git-157)