From 52b6dd9b6cb559bfa55e4b3fbde0d12f6b49f695 Mon Sep 17 00:00:00 2001 From: Johannes Edmeier Date: Fri, 25 Sep 2026 23:50:49 +0200 Subject: [PATCH v1 2/4] Decide at plan time which columns to detoast once per row. set_plan_detoast_reuse, run once a node and its children have their references fixed, records per input the toastable attributes that two or more argument positions of the node's targetlist and quals read (pull_detoast_vars), skipping functions that read only a slice or the size of their argument. Scans, joins on both sides including merge and hash clauses, aggregates and window functions get sets; the executor compiles the argument positions reading them to the EEOP_*_VAR_DETOAST steps. Counting spans one node boundary rather than stopping at the expressions of a single node, which matters as much as sharing within a node does: a large column is as likely to be read once by a scan's filter and once by the node directly above it as it is to be read twice in one place, and a per-node count alone would share neither reference. It reaches that one boundary only: a reference above an intervening join is neither counted nor shared. On the outer side of a join a single reference is enough, since the outer tuple stays put while the join runs its expressions once per inner row. References split between a scan and the node above it add up as well: a column the scan reads once in its filter and a join or aggregate reads through a plain Var of the scan is marked in both, so that the scan makes the copy on its reference and the parent finds it, in the scan slot it reads directly or carried up by the scan's projection with EEOP_ASSIGN_*_VAR_DETOAST; Append and MergeAppend members count as children of the node above them. The ASSIGN variant is emitted only where the result slot's implementation promises the resets, since the copy it carries is borrowed from the source slot rather than owned by the result slot. Parents that store the tuple first (a Hash node, a WindowAgg's tuplestore) are left out, and a plain or sorted Agg reads the first row of each group from a copied tuple and shares from the second row on. Merge clause sides and the arguments of set-returning function calls are compiled as argument positions through ExecInitExprArg. Nothing the planner decides can affect results. pull_multi_detoast_walker enumerates the same argument positions as ExecInitDetoastArg, but only the executor's list carries the rule that a copy must never become an expression result; this one decides where a copy is worth making, so the two drifting apart costs a detoast rather than correctness. The sets are stored on the Plan node, so ExecutorStart does no expression walk of its own, and a node that marks nothing lets ExecInitDetoastArg settle every argument position of its expressions with three null tests. --- src/backend/executor/README | 47 ++++-- src/backend/executor/execExpr.c | 82 +++++++++- src/backend/executor/execExprInterp.c | 77 +++++++++ src/backend/executor/execSRF.c | 6 +- src/backend/executor/nodeMergejoin.c | 4 +- src/backend/jit/llvm/llvmjit_expr.c | 20 +++ src/backend/jit/llvm/llvmjit_types.c | 1 + src/backend/optimizer/plan/planner.c | 1 + src/backend/optimizer/plan/setrefs.c | 221 ++++++++++++++++++++++++++ src/backend/optimizer/util/clauses.c | 191 ++++++++++++++++++++++ src/include/executor/execExpr.h | 8 + src/include/executor/executor.h | 2 + src/include/nodes/plannodes.h | 57 +++++++ src/include/optimizer/optimizer.h | 3 + src/tools/pgindent/typedefs.list | 1 + 15 files changed, 702 insertions(+), 19 deletions(-) diff --git a/src/backend/executor/README b/src/backend/executor/README index 8afe320ca1a..76742bb053e 100644 --- a/src/backend/executor/README +++ b/src/backend/executor/README @@ -289,7 +289,8 @@ Only argument positions read the copy. An argument position is a place where a construct consumes a value and does not return it: the arguments of a function or operator call, the two sides of a scalar array operation, the input of an I/O or array coercion, a field or subscript extraction, the -elements of an array construction, the sides of a row comparison and the +elements of an array construction, the sides of a row comparison, the sides +of a merge clause, the arguments of a set-returning function call and the argument of a hash function built for a hash join or hashed grouping. A Var anywhere else, in particular a bare targetlist entry or an input of CASE, COALESCE, GREATEST, LEAST or NULLIF, all of which can return their input @@ -297,19 +298,47 @@ unchanged, compiles as it always did and yields the stored datum. This is why a detoasted copy can never become the result of an expression and never reaches code that must see the stored form. -Two lists implement that rule: the argument positions themselves, which are -the calls to ExecInitDetoastArg() in execExpr.c, and the functions whose -result depends on the stored representation and which must therefore receive -it, ExecFuncReadsStoredForm() in the same file. +Three lists decide where the copy goes. The first two implement the rule +above and must be kept consistent with it; the third only decides where a +copy pays. + +- The argument positions themselves. ExecInitDetoastArg() is called at each + one in execExpr.c, and pull_multi_detoast_walker() in clauses.c counts + references at the same places when the planner decides. Only the + executor's list carries the rule above: a position wrongly listed there + could return a copy as an expression result. The planner's list decides + where a copy pays, so if the two drift apart an attribute is marked and + unused, or a sharing opportunity is missed. +- Functions that must receive the stored representation because their result +depends on it: ExecFuncReadsStoredForm() in execExpr.c. +- Functions that read only a slice or the size of their argument, which the + planner does not count as references worth sharing: + ExecFuncReadsSliceOrSize() in execExpr.c. A column shared for other + reasons is still handed to one of these, since by then the copy exists and + using it is cheaper than fetching a slice as well. Lifetime. A copy belongs to the tuple the slot holds. A slot implementation promises, through TupleTableSlotOps.resets_detoasted, to release the copies whenever it invalidates tts_values, in its store, clear and materialize paths; an implementation that leaves the field false keeps no copies at all and -simply detoasts per reference. - -Which columns are worth a copy is not decided here. ExecInitDetoastArg() -reads the sets on the Plan node, which nothing fills in yet. +simply detoasts per reference. A projection of a marked column carries the +copy into the result slot alongside the stored datum, so a parent node reading +that column as an argument finds it. The copy and the stored datum are both +borrowed from the source slot and so share its lifetime; materializing the +result slot therefore drops the copy rather than taking ownership of it, since +a materialized slot may outlive its source. + +The planner decides only where this pays, in set_plan_detoast_reuse() in +setrefs.c, which records per node the attributes two or more argument +positions read, and marks a single reference where the node re-evaluates its +expressions against an unchanging tuple, such as the outer side of a join. +References are counted across one node boundary as well as within a node: if a +scan reads a column once and the node directly above it reads that column once +as well, the column is marked on both. The counting reaches no further, so a +reference above an intervening join is neither counted nor shared; reaching +further would mean marking the column on every node in between and carrying +the copy up through each of their projections. Nothing about those sets +affects results: a wrong decision costs a detoast, never correctness. Memory Management ----------------- diff --git a/src/backend/executor/execExpr.c b/src/backend/executor/execExpr.c index 015c89464d5..30305457b54 100644 --- a/src/backend/executor/execExpr.c +++ b/src/backend/executor/execExpr.c @@ -73,6 +73,8 @@ typedef struct ExprSetupInfo } ExprSetupInfo; static void ExecReadyExpr(ExprState *state); +static ExprState *ExecInitExprInternal(Expr *node, PlanState *parent, + bool detoast_arg); static bool ExecPushDetoastArgStep(Expr *expr, ExprState *state, Datum *resv, bool *resnull); static inline void ExecInitDetoastArg(Expr *arg, ExprState *state, @@ -149,6 +151,44 @@ static void ExecInitJsonCoercion(ExprState *state, JsonReturning *returning, */ ExprState * ExecInitExpr(Expr *node, PlanState *parent) +{ + return ExecInitExprInternal(node, parent, false); +} + +/* + * As ExecInitExpr, for an expression whose result the caller consumes without + * returning it, so that the whole expression is an argument position in the + * sense of ExecInitDetoastArg(). Merge clause sides and the arguments of a + * set-returning function call are compiled this way. + */ +ExprState * +ExecInitExprArg(Expr *node, PlanState *parent) +{ + return ExecInitExprInternal(node, parent, true); +} + +/* + * Call ExecInitExprArg() on a list of expressions, return a list of + * ExprStates. + */ +List * +ExecInitExprArgList(List *nodes, PlanState *parent) +{ + List *result = NIL; + ListCell *lc; + + foreach(lc, nodes) + { + Expr *e = lfirst(lc); + + result = lappend(result, ExecInitExprArg(e, parent)); + } + + return result; +} + +static ExprState * +ExecInitExprInternal(Expr *node, PlanState *parent, bool detoast_arg) { ExprState *state; ExprEvalStep scratch = {0}; @@ -167,7 +207,10 @@ ExecInitExpr(Expr *node, PlanState *parent) ExecCreateExprSetupSteps(state, (Node *) node); /* Compile the expression proper */ - ExecInitExprRec(node, state, &state->resvalue, &state->resnull); + if (detoast_arg) + ExecInitDetoastArg(node, state, &state->resvalue, &state->resnull); + else + ExecInitExprRec(node, state, &state->resvalue, &state->resnull); /* Finally, append a DONE step */ scratch.opcode = EEOP_DONE_RETURN; @@ -385,6 +428,7 @@ ExecBuildProjectionInfo(List *targetList, ExprState *state; ExprEvalStep scratch = {0}; ListCell *lc; + bool carry_detoast; projInfo->pi_exprContext = econtext; /* We embed ExprState into ProjectionInfo instead of doing extra palloc */ @@ -396,6 +440,16 @@ ExecBuildProjectionInfo(List *targetList, state->resultslot = slot; + /* + * A column the node detoasts once per row can have its copy + * carried into the result slot beside the stored datum. The copy + * is borrowed from the source slot rather than owned by the result + * slot, so the result slot must be one that releases the copies + * when it invalidates tts_values. See ExecEvalAssignVarDetoast(). + */ + carry_detoast = detoast_reuse && parent != NULL && parent->plan != NULL && + slot->tts_ops->resets_detoasted; + /* Insert setup steps as needed */ ExecCreateExprSetupSteps(state, (Node *) targetList); @@ -444,18 +498,31 @@ ExecBuildProjectionInfo(List *targetList, if (isSafeVar) { /* - * Fast-path: just generate an EEOP_ASSIGN_*_VAR step + * Fast-path: just generate an EEOP_ASSIGN_*_VAR step. A column + * the node detoasts once per row gets the variant that carries + * the copy into the result slot alongside the stored datum, so + * that a parent reading the column as an argument finds it. */ switch (variable->varno) { case INNER_VAR: /* get the tuple from the inner node */ - scratch.opcode = EEOP_ASSIGN_INNER_VAR; + if (carry_detoast && + bms_is_member(attnum, + parent->plan->detoast_reuse_inner)) + scratch.opcode = EEOP_ASSIGN_INNER_VAR_DETOAST; + else + scratch.opcode = EEOP_ASSIGN_INNER_VAR; break; case OUTER_VAR: /* get the tuple from the outer node */ - scratch.opcode = EEOP_ASSIGN_OUTER_VAR; + if (carry_detoast && + bms_is_member(attnum, + parent->plan->detoast_reuse_outer)) + scratch.opcode = EEOP_ASSIGN_OUTER_VAR_DETOAST; + else + scratch.opcode = EEOP_ASSIGN_OUTER_VAR; break; /* INDEX_VAR is handled by default case */ @@ -469,7 +536,12 @@ ExecBuildProjectionInfo(List *targetList, switch (variable->varreturningtype) { case VAR_RETURNING_DEFAULT: - scratch.opcode = EEOP_ASSIGN_SCAN_VAR; + if (carry_detoast && + bms_is_member(attnum, + parent->plan->detoast_reuse_scan)) + scratch.opcode = EEOP_ASSIGN_SCAN_VAR_DETOAST; + else + scratch.opcode = EEOP_ASSIGN_SCAN_VAR; break; case VAR_RETURNING_OLD: scratch.opcode = EEOP_ASSIGN_OLD_VAR; diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c index 97d140e6e5f..90236d68718 100644 --- a/src/backend/executor/execExprInterp.c +++ b/src/backend/executor/execExprInterp.c @@ -535,6 +535,9 @@ ExecInterpExpr(ExprState *state, ExprContext *econtext, bool *isnull) &&CASE_EEOP_ASSIGN_SCAN_VAR, &&CASE_EEOP_ASSIGN_OLD_VAR, &&CASE_EEOP_ASSIGN_NEW_VAR, + &&CASE_EEOP_ASSIGN_INNER_VAR_DETOAST, + &&CASE_EEOP_ASSIGN_OUTER_VAR_DETOAST, + &&CASE_EEOP_ASSIGN_SCAN_VAR_DETOAST, &&CASE_EEOP_ASSIGN_TMP, &&CASE_EEOP_ASSIGN_TMP_MAKE_RO, &&CASE_EEOP_CONST, @@ -895,6 +898,26 @@ ExecInterpExpr(ExprState *state, ExprContext *econtext, bool *isnull) EEO_NEXT(); } + EEO_CASE(EEOP_ASSIGN_INNER_VAR_DETOAST) + { + ExecEvalAssignVarDetoast(state, op, econtext, innerslot); + + EEO_NEXT(); + } + + EEO_CASE(EEOP_ASSIGN_OUTER_VAR_DETOAST) + { + ExecEvalAssignVarDetoast(state, op, econtext, outerslot); + + EEO_NEXT(); + } + + EEO_CASE(EEOP_ASSIGN_SCAN_VAR_DETOAST) + { + ExecEvalAssignVarDetoast(state, op, econtext, scanslot); + + EEO_NEXT(); + } EEO_CASE(EEOP_ASSIGN_OLD_VAR) { @@ -5808,6 +5831,60 @@ ExecEvalVarDetoast(ExprState *state, ExprEvalStep *op, ExprContext *econtext, ExecEvalVarDetoastInline(op, slot); } +/* + * Projection of a column the node detoasts once per row: the stored datum + * goes into the result slot as usual, and the copy alongside it, so a parent + * reading the column as an argument finds it rather than making its own. + * + * Both are borrowed, not owned: the result slot takes the source slot's + * pointers, and the detoasted copy stays in the source slot's detoast + * context, exactly as the stored datum stays in the source slot's tuple. + * The two therefore have the same lifetime, and any use of the result slot + * that is safe for the stored datum is safe for the copy. Breaking that + * dependency is materialization, which drops the borrowed copies rather than + * copying them (see tts_virtual_materialize()), so a slot that outlives its + * source never holds a pointer into it. ExecBuildProjectionInfo() emits + * this step only for a result slot whose implementation promises the resets + * that make keeping copies safe at all. + */ +void +ExecEvalAssignVarDetoast(ExprState *state, ExprEvalStep *op, + ExprContext *econtext, TupleTableSlot *slot) +{ + TupleTableSlot *resultslot = state->resultslot; + int resultnum = op->d.assign_var.resultnum; + int attnum = op->d.assign_var.attnum; + + Assert(attnum >= 0 && attnum < slot->tts_nvalid); + Assert(resultnum >= 0 && resultnum < resultslot->tts_tupleDescriptor->natts); + Assert(resultslot->tts_ops->resets_detoasted); + resultslot->tts_values[resultnum] = slot->tts_values[attnum]; + resultslot->tts_isnull[resultnum] = slot->tts_isnull[attnum]; + + if (slot->tts_detoasted != NULL && slot->tts_detoasted[attnum] != (Datum) 0) + { + if (unlikely(resultslot->tts_detoast_cxt == NULL)) + resultslot->tts_detoast_cxt = + GenerationContextCreate(resultslot->tts_mcxt, + "detoasted slot values", + ALLOCSET_DEFAULT_SIZES); + if (resultslot->tts_detoasted == NULL) + resultslot->tts_detoasted = + MemoryContextAllocZero(resultslot->tts_detoast_cxt, + resultslot->tts_tupleDescriptor->natts * + sizeof(Datum)); + resultslot->tts_detoasted[resultnum] = slot->tts_detoasted[attnum]; +#ifdef USE_ASSERT_CHECKING + if (resultslot->tts_detoast_src == NULL) + resultslot->tts_detoast_src = + MemoryContextAllocZero(resultslot->tts_detoast_cxt, + resultslot->tts_tupleDescriptor->natts * + sizeof(Datum)); + resultslot->tts_detoast_src[resultnum] = resultslot->tts_values[resultnum]; +#endif + } +} + void ExecEvalSysVar(ExprState *state, ExprEvalStep *op, ExprContext *econtext, TupleTableSlot *slot) diff --git a/src/backend/executor/execSRF.c b/src/backend/executor/execSRF.c index f839f3bbde7..1976474c986 100644 --- a/src/backend/executor/execSRF.c +++ b/src/backend/executor/execSRF.c @@ -77,7 +77,7 @@ ExecInitTableFunctionResult(Expr *expr, FuncExpr *func = (FuncExpr *) expr; state->funcReturnsSet = func->funcretset; - state->args = ExecInitExprList(func->args, parent); + state->args = ExecInitExprArgList(func->args, parent); init_sexpr(func->funcid, func->inputcollid, expr, state, parent, econtext->ecxt_per_query_memory, func->funcretset, false); @@ -460,7 +460,7 @@ ExecInitFunctionResultSet(Expr *expr, { FuncExpr *func = (FuncExpr *) expr; - state->args = ExecInitExprList(func->args, parent); + state->args = ExecInitExprArgList(func->args, parent); init_sexpr(func->funcid, func->inputcollid, expr, state, parent, econtext->ecxt_per_query_memory, true, true); } @@ -468,7 +468,7 @@ ExecInitFunctionResultSet(Expr *expr, { OpExpr *op = (OpExpr *) expr; - state->args = ExecInitExprList(op->args, parent); + state->args = ExecInitExprArgList(op->args, parent); init_sexpr(op->opfuncid, op->inputcollid, expr, state, parent, econtext->ecxt_per_query_memory, true, true); } diff --git a/src/backend/executor/nodeMergejoin.c b/src/backend/executor/nodeMergejoin.c index 9e14ac78613..c9f6011b78e 100644 --- a/src/backend/executor/nodeMergejoin.c +++ b/src/backend/executor/nodeMergejoin.c @@ -208,8 +208,8 @@ MJExamineQuals(List *mergeclauses, /* * Prepare the input expressions for execution. */ - clause->lexpr = ExecInitExpr((Expr *) linitial(qual->args), parent); - clause->rexpr = ExecInitExpr((Expr *) lsecond(qual->args), parent); + clause->lexpr = ExecInitExprArg((Expr *) linitial(qual->args), parent); + clause->rexpr = ExecInitExprArg((Expr *) lsecond(qual->args), parent); /* Set up sort support data */ clause->ssup.ssup_cxt = CurrentMemoryContext; diff --git a/src/backend/jit/llvm/llvmjit_expr.c b/src/backend/jit/llvm/llvmjit_expr.c index 288f8b7f2fc..d172e7a42c3 100644 --- a/src/backend/jit/llvm/llvmjit_expr.c +++ b/src/backend/jit/llvm/llvmjit_expr.c @@ -508,6 +508,26 @@ llvm_compile_expr(ExprState *state) break; } + case EEOP_ASSIGN_INNER_VAR_DETOAST: + case EEOP_ASSIGN_OUTER_VAR_DETOAST: + case EEOP_ASSIGN_SCAN_VAR_DETOAST: + { + LLVMValueRef v_slot; + + if (opcode == EEOP_ASSIGN_INNER_VAR_DETOAST) + v_slot = v_innerslot; + else if (opcode == EEOP_ASSIGN_OUTER_VAR_DETOAST) + v_slot = v_outerslot; + else + v_slot = v_scanslot; + + build_EvalXFunc(b, mod, "ExecEvalAssignVarDetoast", + v_state, op, v_econtext, v_slot); + + LLVMBuildBr(b, opblocks[opno + 1]); + break; + } + case EEOP_INNER_SYSVAR: case EEOP_OUTER_SYSVAR: case EEOP_SCAN_SYSVAR: diff --git a/src/backend/jit/llvm/llvmjit_types.c b/src/backend/jit/llvm/llvmjit_types.c index 034f0193e98..a825ff8e9a6 100644 --- a/src/backend/jit/llvm/llvmjit_types.c +++ b/src/backend/jit/llvm/llvmjit_types.c @@ -170,6 +170,7 @@ void *referenced_functions[] = ExecEvalSubPlan, ExecEvalSysVar, ExecEvalVarDetoast, + ExecEvalAssignVarDetoast, ExecEvalWholeRowVar, ExecEvalXmlExpr, ExecEvalJsonConstructor, diff --git a/src/backend/optimizer/plan/planner.c b/src/backend/optimizer/plan/planner.c index 9458efab480..a028975ab66 100644 --- a/src/backend/optimizer/plan/planner.c +++ b/src/backend/optimizer/plan/planner.c @@ -670,6 +670,7 @@ standard_planner(Query *parse, const char *query_string, int cursorOptions, result->hasReturning = (parse->returningList != NIL); result->hasModifyingCTE = parse->hasModifyingCTE; result->canSetTag = parse->canSetTag; + result->detoastReuse = detoast_reuse; result->transientPlan = glob->transientPlan; result->dependsOnRole = glob->dependsOnRole; result->parallelModeNeeded = glob->parallelModeNeeded; diff --git a/src/backend/optimizer/plan/setrefs.c b/src/backend/optimizer/plan/setrefs.c index 8a641402a96..445b0451833 100644 --- a/src/backend/optimizer/plan/setrefs.c +++ b/src/backend/optimizer/plan/setrefs.c @@ -26,9 +26,11 @@ #include "optimizer/subselect.h" #include "optimizer/tlist.h" #include "parser/parse_relation.h" +#include "parser/parsetree.h" #include "rewrite/rewriteManip.h" #include "tcop/utility.h" #include "utils/syscache.h" +#include "utils/typcache.h" typedef enum @@ -134,6 +136,8 @@ static bool flatten_rtes_walker(Node *node, flatten_rtes_walker_context *cxt); static void add_rte_to_flat_rtable(PlannerGlobal *glob, List *rteperminfos, RangeTblEntry *rte); static Plan *set_plan_refs(PlannerInfo *root, Plan *plan, int rtoffset); +static Index scan_tuple_varno(Plan *plan); +static void set_plan_detoast_reuse(Plan *plan); static Plan *set_indexonlyscan_references(PlannerInfo *root, IndexOnlyScan *plan, int rtoffset); @@ -630,6 +634,214 @@ add_rte_to_flat_rtable(PlannerGlobal *glob, List *rteperminfos, } } +/* Detoast a toasted column once per row when several expressions reference it */ + +/* + * toastable_type + * Can values of this type be stored out of line or compressed? The + * type cache answers without a catalog fetch, unlike TypeIsToastable(); + * only varlena types have anything but PLAIN storage. + */ +static bool +toastable_type(Oid typid) +{ + return lookup_type_cache(typid, 0)->typstorage != TYPSTORAGE_PLAIN; +} + +/* + * scan_tuple_varno + * The varno the expressions of a scan node use for its scan tuple: its + * own range table index, or INDEX_VAR for the scans whose tuple has a + * shape of its own (see ScanUsesIndexVar). Both spaces never occur in + * one node's targetlist and qual, but naming the right one keeps the + * attribute numbers we record in the space the executor reads them in. + */ +static Index +scan_tuple_varno(Plan *plan) +{ + return ScanUsesIndexVar(plan) ? INDEX_VAR : ((Scan *) plan)->scanrelid; +} + +/* + * join_side_quals + * The expressions a join evaluates besides its targetlist: quals, join + * quals and, being argument positions as well, the merge or hash + * clauses. The caller frees the list. + */ +static List * +join_side_quals(Join *join) +{ + List *quals = list_concat_copy(join->joinqual, join->plan.qual); + + if (IsA(join, MergeJoin)) + quals = list_concat(quals, ((MergeJoin *) join)->mergeclauses); + else if (IsA(join, HashJoin)) + quals = list_concat(quals, ((HashJoin *) join)->hashclauses); + return quals; +} + +/* + * expand_append_members + * Add to result the plans a node receives its input from through child, + * looking through Append and MergeAppend, which hand their members' + * slots up unchanged. + */ +static List * +expand_append_members(Plan *child, List *result) +{ + if (child == NULL) + return result; + if (IsA(child, Append)) + { + foreach_ptr(Plan, member, ((Append *) child)->appendplans) + result = expand_append_members(member, result); + return result; + } + if (IsA(child, MergeAppend)) + { + foreach_ptr(Plan, member, ((MergeAppend *) child)->mergeplans) + result = expand_append_members(member, result); + return result; + } + return lappend(result, child); +} + +/* + * input_detoast_reuse_attrs + * The attributes of one input of a node that its expressions may detoast + * once per row: toastable attributes that two or more argument positions + * of the targetlist and quals read, or one when single_ref_ok (the outer + * side of a join, whose tuple stays put while the join runs its + * expressions once per inner row, so the copy serves all of them). + * + * References split between this node and a scan it reads directly add up as + * well. If the scan reads a column, and this node reads that same column + * through a plain Var of the scan in the child's targetlist, the column is + * marked on both: the scan makes the copy on its own reference, and this node + * finds it, either in the scan slot it reads directly or carried up by the + * scan's projection. child is that input plan; an Append or MergeAppend of + * scans counts as its members. Only + * that direct child is examined, so a scan further down, under a join or + * another intermediate node, is not reached and its references do not add up + * with the ones here; that costs a detoast, not a wrong answer. + * Parents that store the tuple first (a Hash node, a WindowAgg's tuplestore) + * pass NULL, since the copy does not reach them; a plain or sorted Agg reads + * the first row of each group from a copied tuple and shares from the second + * row on. A child output column that is a constant or a row built in memory + * is never toasted and is skipped. + */ +static Bitmapset * +input_detoast_reuse_attrs(List *tlist, List *quals, Index varno, + bool single_ref_ok, Plan *child) +{ + Bitmapset *multi; + List *vars = pull_detoast_vars(tlist, quals, varno, &multi); + List *cands = NIL; + Bitmapset *attrs = NULL; + List *members; + ListCell *lc; + + foreach(lc, vars) + { + Var *var = (Var *) lfirst(lc); + TargetEntry *tle; + + if (!toastable_type(var->vartype)) + continue; + if (child != NULL && + (tle = get_tle_by_resno(child->targetlist, var->varattno)) != NULL && + (IsA(tle->expr, Const) || IsA(tle->expr, RowExpr))) + continue; + cands = lappend(cands, var); + if (single_ref_ok || bms_is_member(var->varattno, multi)) + attrs = bms_add_member(attrs, var->varattno); + } + + members = expand_append_members(child, NIL); + foreach_ptr(Plan, member, members) + { + Bitmapset *cmulti; + List *cvars; + Bitmapset *cseen = NULL; + ListCell *clc; + + if (!IsScanPlan(member) || cands == NIL) + continue; + cvars = pull_detoast_vars(member->targetlist, member->qual, + scan_tuple_varno(member), &cmulti); + foreach(clc, cvars) + cseen = bms_add_member(cseen, ((Var *) lfirst(clc))->varattno); + + foreach(clc, cands) + { + Var *var = (Var *) lfirst(clc); + TargetEntry *tle = get_tle_by_resno(member->targetlist, + var->varattno); + Node *expr; + + if (tle == NULL) + continue; + expr = (Node *) tle->expr; + while (IsA(expr, RelabelType)) + expr = (Node *) ((RelabelType *) expr)->arg; + if (!IsA(expr, Var) || + !bms_is_member(((Var *) expr)->varattno, cseen)) + continue; + member->detoast_reuse_scan = + bms_add_member(member->detoast_reuse_scan, + ((Var *) expr)->varattno); + attrs = bms_add_member(attrs, var->varattno); + } + list_free(cvars); + bms_free(cmulti); + bms_free(cseen); + } + list_free(members); + list_free(cands); + list_free(vars); + bms_free(multi); + return attrs; +} + +/* + * set_plan_detoast_reuse + * Record, once a node and its children have their references fixed, + * which input attributes the executor may detoast once per row (see + * input_detoast_reuse_attrs). Nothing about the sets affects results: + * they only decide where the copy is worth making. + */ +static void +set_plan_detoast_reuse(Plan *plan) +{ + if (!detoast_reuse) + return; + + if (IsScanPlan(plan)) + plan->detoast_reuse_scan = + input_detoast_reuse_attrs(plan->targetlist, plan->qual, + scan_tuple_varno(plan), false, NULL); + else if (IsA(plan, NestLoop) || IsA(plan, MergeJoin) || IsA(plan, HashJoin)) + { + List *quals = join_side_quals((Join *) plan); + + plan->detoast_reuse_outer = + input_detoast_reuse_attrs(plan->targetlist, quals, OUTER_VAR, true, + plan->lefttree); + plan->detoast_reuse_inner = + input_detoast_reuse_attrs(plan->targetlist, quals, INNER_VAR, false, + plan->righttree); + list_free(quals); + } + else if (IsA(plan, Agg)) + plan->detoast_reuse_outer = + input_detoast_reuse_attrs(plan->targetlist, plan->qual, OUTER_VAR, + false, plan->lefttree); + else if (IsA(plan, WindowAgg)) + plan->detoast_reuse_outer = + input_detoast_reuse_attrs(plan->targetlist, plan->qual, OUTER_VAR, + false, NULL); +} + /* * set_plan_refs: recurse through the Plan nodes of a single subquery level */ @@ -1357,6 +1569,12 @@ set_plan_refs(PlannerInfo *root, Plan *plan, int rtoffset) plan->lefttree = set_plan_refs(root, plan->lefttree, rtoffset); plan->righttree = set_plan_refs(root, plan->righttree, rtoffset); + /* + * With the node's and its children's expressions final, record what the + * executor may detoast once per row. + */ + set_plan_detoast_reuse(plan); + return plan; } @@ -1430,6 +1648,8 @@ set_indexonlyscan_references(PlannerInfo *root, pfree(index_itlist); + set_plan_detoast_reuse((Plan *) plan); + return (Plan *) plan; } @@ -1484,6 +1704,7 @@ set_subqueryscan_references(PlannerInfo *root, plan->scan.plan.qual = fix_scan_list(root, plan->scan.plan.qual, rtoffset, NUM_EXEC_QUAL((Plan *) plan)); + set_plan_detoast_reuse((Plan *) plan); result = (Plan *) plan; } diff --git a/src/backend/optimizer/util/clauses.c b/src/backend/optimizer/util/clauses.c index 3e1f210652d..a15c83c723d 100644 --- a/src/backend/optimizer/util/clauses.c +++ b/src/backend/optimizer/util/clauses.c @@ -6829,3 +6829,194 @@ make_SAOP_expr(Oid oper, Node *leftexpr, Oid coltype, Oid arraycollid, return saopexpr; } + +/***************************************************************************** + * Detoasting references to scan-slot Vars + *****************************************************************************/ + +typedef struct +{ + Index varno; /* only Vars of this varno count; 0 = any */ + Bitmapset *seen; /* attnos seen in a detoasting position */ + Bitmapset *multi; /* attnos seen in two or more */ + List *vars; /* one Var per attno in seen */ +} pull_multi_detoast_context; + +static bool pull_multi_detoast_walker(Node *node, + pull_multi_detoast_context *context); + +static Var * +strip_relabel_var(Node *arg, Index varno) +{ + while (IsA(arg, RelabelType)) + arg = (Node *) ((RelabelType *) arg)->arg; + if (IsA(arg, Var) && ((Var *) arg)->varattno > 0 && + ((Var *) arg)->varreturningtype == VAR_RETURNING_DEFAULT && + (varno == 0 || ((Var *) arg)->varno == varno)) + return (Var *) arg; + return NULL; +} + +static void +pull_multi_detoast_count(Node *arg, pull_multi_detoast_context *context) +{ + Var *var = strip_relabel_var(arg, context->varno); + + if (var == NULL) + { + pull_multi_detoast_walker(arg, context); + return; + } + if (bms_is_member(var->varattno, context->seen)) + context->multi = bms_add_member(context->multi, var->varattno); + else + { + context->seen = bms_add_member(context->seen, var->varattno); + context->vars = lappend(context->vars, var); + } +} + +/* + * Treat each argument of a function-like node as a detoasting reference when + * it is a plain Var (possibly relabeled); recurse into anything else. + */ +static void +pull_multi_detoast_args(List *args, bool detoasts, + pull_multi_detoast_context *context) +{ + ListCell *lc; + + foreach(lc, args) + { + Node *arg = (Node *) lfirst(lc); + + if (detoasts) + pull_multi_detoast_count(arg, context); + else + pull_multi_detoast_walker(arg, context); + } +} + +static bool +pull_multi_detoast_walker(Node *node, pull_multi_detoast_context *context) +{ + if (node == NULL) + return false; + + switch (nodeTag(node)) + { + case T_Var: + /* a Var reached here is passed along whole, not detoasted */ + return false; + case T_FuncExpr: + { + FuncExpr *f = (FuncExpr *) node; + + /* + * Functions that inspect the stored form are compiled to + * receive the stored datum (see ExecInitFunc), so like slice + * readers they do not count as detoasting references. + */ + pull_multi_detoast_args(f->args, + !ExecFuncReadsStoredForm(f->funcid) && + !ExecFuncReadsSliceOrSize(f->funcid), + context); + return false; + } + case T_OpExpr: + case T_DistinctExpr: + pull_multi_detoast_args(((OpExpr *) node)->args, true, context); + return false; + case T_ScalarArrayOpExpr: + pull_multi_detoast_args(((ScalarArrayOpExpr *) node)->args, true, + context); + return false; + case T_CoerceViaIO: + pull_multi_detoast_count((Node *) ((CoerceViaIO *) node)->arg, + context); + return false; + case T_ArrayCoerceExpr: + pull_multi_detoast_count((Node *) ((ArrayCoerceExpr *) node)->arg, + context); + pull_multi_detoast_walker((Node *) ((ArrayCoerceExpr *) node)->elemexpr, + context); + return false; + case T_FieldSelect: + pull_multi_detoast_count((Node *) ((FieldSelect *) node)->arg, + context); + return false; + case T_SubscriptingRef: + { + SubscriptingRef *sbsref = (SubscriptingRef *) node; + + pull_multi_detoast_count((Node *) sbsref->refexpr, context); + pull_multi_detoast_walker((Node *) sbsref->refupperindexpr, + context); + pull_multi_detoast_walker((Node *) sbsref->reflowerindexpr, + context); + pull_multi_detoast_walker((Node *) sbsref->refassgnexpr, + context); + return false; + } + case T_ArrayExpr: + pull_multi_detoast_args(((ArrayExpr *) node)->elements, true, + context); + return false; + case T_RowCompareExpr: + pull_multi_detoast_args(((RowCompareExpr *) node)->largs, true, + context); + pull_multi_detoast_args(((RowCompareExpr *) node)->rargs, true, + context); + return false; + default: + + /* + * Everything else (CASE, COALESCE, GREATEST/LEAST, NULLIF, ROW(), + * boolean operators, NullTest, TargetEntry, ...) hands the datum + * on without looking inside it, or returns it unchanged; only + * what it feeds into can detoast. The executor has the matching + * list, the places ExecInitDetoastArg is used in execExpr.c, and + * that one is what keeps a copy out of positions that return + * their input; this one only decides where a copy is worth + * making, so the two drifting apart costs a detoast rather than + * correctness. + */ + return expression_tree_walker(node, pull_multi_detoast_walker, + context); + } +} + +/* + * pull_detoast_vars + * Find the scan-slot Vars that a plan node's targetlist and qual would + * detoast: one Var per attribute read in an argument position, with + * *multi holding the attributes read in two or more. + * + * A reference counts when the Var is a direct argument of a function-like + * node that reads the whole value and does not return it: function and + * operator calls, casts through I/O functions, field and subscript access, + * array construction and row comparison. Bare Vars, Vars under constructs + * that may return them unchanged (CASE, COALESCE, GREATEST/LEAST, NULLIF), + * and Vars passed to functions known to read only a slice or the size of + * their argument, or to a function that inspects the stored form, do not + * count. Only Vars with the given varno count (0 means any, for scan nodes; + * OUTER_VAR or INNER_VAR for joins and aggregates). The caller checks + * toastability. + */ +List * +pull_detoast_vars(List *targetlist, List *qual, Index varno, Bitmapset **multi) +{ + pull_multi_detoast_context context; + + context.varno = varno; + context.seen = NULL; + context.multi = NULL; + context.vars = NIL; + + pull_multi_detoast_walker((Node *) targetlist, &context); + pull_multi_detoast_walker((Node *) qual, &context); + + bms_free(context.seen); + *multi = context.multi; + return context.vars; +} diff --git a/src/include/executor/execExpr.h b/src/include/executor/execExpr.h index 469fd4be4ae..98d7b30744e 100644 --- a/src/include/executor/execExpr.h +++ b/src/include/executor/execExpr.h @@ -116,6 +116,11 @@ typedef enum ExprEvalOp EEOP_ASSIGN_OLD_VAR, EEOP_ASSIGN_NEW_VAR, + /* same, carrying a detoasted copy of the column into the result slot */ + EEOP_ASSIGN_INNER_VAR_DETOAST, + EEOP_ASSIGN_OUTER_VAR_DETOAST, + EEOP_ASSIGN_SCAN_VAR_DETOAST, + /* assign ExprState's resvalue/resnull to a column of its resultslot */ EEOP_ASSIGN_TMP, /* ditto, applying MakeExpandedObjectReadOnly() */ @@ -918,6 +923,9 @@ extern void ExecEvalSysVar(ExprState *state, ExprEvalStep *op, ExprContext *econtext, TupleTableSlot *slot); extern void ExecEvalVarDetoast(ExprState *state, ExprEvalStep *op, ExprContext *econtext, TupleTableSlot *slot); +extern void ExecEvalAssignVarDetoast(ExprState *state, ExprEvalStep *op, + ExprContext *econtext, + TupleTableSlot *slot); extern void ExecAggInitGroup(AggState *aggstate, AggStatePerTrans pertrans, AggStatePerGroup pergroup, ExprContext *aggcontext); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index f073e54a4e0..3d55a7fbf14 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -94,6 +94,8 @@ extern PGDLLIMPORT ExecutorEnd_hook_type ExecutorEnd_hook; /* GUC parameter */ extern PGDLLIMPORT bool detoast_reuse; +extern ExprState *ExecInitExprArg(Expr *node, PlanState *parent); +extern List *ExecInitExprArgList(List *nodes, PlanState *parent); extern bool ExecFuncReadsStoredForm(Oid funcid); extern bool ExecFuncReadsSliceOrSize(Oid funcid); diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h index b2f54d3a93f..49d8727755b 100644 --- a/src/include/nodes/plannodes.h +++ b/src/include/nodes/plannodes.h @@ -562,6 +562,38 @@ typedef struct Scan Index scanrelid; } Scan; +/* + * Is this plan node a Scan (or a type derived from Scan)? Executor states + * of several non-scan nodes (Agg, Sort, Material, ...) embed a ScanState, so + * code reached through one cannot assume the plan is a Scan without asking. + */ +static inline bool +IsScanPlan(const Plan *plan) +{ + switch (nodeTag(plan)) + { + case T_SeqScan: + case T_SampleScan: + case T_IndexScan: + case T_IndexOnlyScan: + case T_BitmapHeapScan: + case T_TidScan: + case T_TidRangeScan: + case T_SubqueryScan: + case T_FunctionScan: + case T_TableFuncScan: + case T_ValuesScan: + case T_CteScan: + case T_NamedTuplestoreScan: + case T_WorkTableScan: + case T_ForeignScan: + case T_CustomScan: + return true; + default: + return false; + } +} + /* ---------------- * sequential scan node * ---------------- @@ -971,6 +1003,31 @@ typedef struct CustomScan const struct CustomScanMethods *methods; } CustomScan; +/* + * Does this scan's targetlist refer to the scan tuple through INDEX_VAR? + * That is the case when the scan tuple has a shape of its own rather than + * the table's: index-only scans, and foreign or custom scans with a scan + * targetlist (always so when scanrelid is 0, since there is no table). The + * executor picks the projection varno by the same rule. + */ +static inline bool +ScanUsesIndexVar(const Plan *plan) +{ + switch (nodeTag(plan)) + { + case T_IndexOnlyScan: + return true; + case T_ForeignScan: + return ((const ForeignScan *) plan)->fdw_scan_tlist != NIL || + ((const Scan *) plan)->scanrelid == 0; + case T_CustomScan: + return ((const CustomScan *) plan)->custom_scan_tlist != NIL || + ((const Scan *) plan)->scanrelid == 0; + default: + return false; + } +} + /* * ========== * Join nodes diff --git a/src/include/optimizer/optimizer.h b/src/include/optimizer/optimizer.h index cb6241e2bdd..f2e6a0139a0 100644 --- a/src/include/optimizer/optimizer.h +++ b/src/include/optimizer/optimizer.h @@ -141,6 +141,9 @@ typedef enum extern bool contain_mutable_functions(Node *clause); extern bool contain_mutable_functions_after_planning(Expr *expr); extern bool contain_volatile_functions(Node *clause); +extern PGDLLIMPORT bool detoast_reuse; +extern List *pull_detoast_vars(List *targetlist, List *qual, Index varno, + Bitmapset **multi); extern bool contain_volatile_functions_after_planning(Expr *expr); extern bool contain_volatile_functions_not_nextval(Node *clause); diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 656f1f60862..d6d5a98fedc 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -2106,6 +2106,7 @@ PTEntryArray PTIterationArray PTOKEN_PRIVILEGES PTOKEN_USER +pull_multi_detoast_context PUTENVPROC PVIndStats PVIndVacStatus -- 2.55.0