From 62b6f89f042eaad86f71af3127ae308906f76268 Mon Sep 17 00:00:00 2001 From: Johannes Edmeier Date: Fri, 25 Sep 2026 23:49:09 +0200 Subject: [PATCH v1 1/4] Executor support for detoasting a column once per row. Each operator or function applied to a toasted column fetches and decompresses the value again: a WHERE clause with two jsonb predicates on the same document reads its toast chunks twice. Scanning 2000 rows whose jsonb column is stored out of line, with two operators on that column, reads 6013 buffers where it read 12013 before. This adds the executor side of detoasting such a column once per row. What it does is driven by the per-node sets that the planner fills in a later commit, so on its own this commit changes nothing. The copy is kept beside the slot, in tts_detoasted, allocated in a per-slot memory context that the slot implementation resets whenever tts_values is invalidated, in every store, clear and materialize path. Declaring that is what the new TupleTableSlotOps.resets_detoasted is for: the four in-tree implementations set it, and one that leaves it false, as an implementation predating the field does, keeps no copies and detoasts per reference as before. tts_values keeps the stored datum, so everything that copies, stores or projects the column unchanged, or inspects its stored form, is unaffected. An assert-only tts_detoast_src records the tts_values entry each copy was made from, and slot_detoast_attr() checks it before handing a copy out, so an implementation that declares the resets and then misses one is caught on the first cassert run rather than by a wrong answer much later. A matching address proves nothing on its own, since an address can be reused; it catches the mistake an implementation actually makes. Only argument positions of function-like constructs read the copy, that is places where a construct consumes a value and cannot return it: ExecInitDetoastArg is called at each one, and compiles a plain Var of a marked attribute to EEOP_SCAN_VAR_DETOAST (or the INNER/OUTER variant), which detoasts on first use and returns the copy afterwards. A Var anywhere else, in particular under CASE, COALESCE, GREATEST/LEAST or NULLIF, all of which can return their input, or as a bare targetlist entry, compiles as before. That is what keeps a detoasted value from becoming the result of an expression. pg_column_size, pg_column_compression and pg_column_toast_chunk_id must see the stored form and are excluded. ExecInitDetoastArg runs at every argument position of every expression compiled, and expressions are compiled at every execution, so it settles the common case inline and pushes the step itself rather than letting the ordinary one be compiled and then changing it. A copy lives in a context of the slot's own until the slot is refilled, rather than in the per-tuple context that would have freed it at the end of the row, so a plan holding several wide values in several slots holds them somewhat longer than before. The developer option detoast_reuse switches the whole thing off. --- doc/src/sgml/config.sgml | 26 +++ src/backend/executor/README | 44 +++++ src/backend/executor/execExpr.c | 225 +++++++++++++++++++--- src/backend/executor/execExprInterp.c | 153 +++++++++++++++ src/backend/executor/execTuples.c | 57 +++++- src/backend/jit/llvm/llvmjit_expr.c | 20 ++ src/backend/jit/llvm/llvmjit_types.c | 1 + src/backend/utils/misc/guc_parameters.dat | 7 + src/backend/utils/misc/guc_tables.c | 1 + src/include/executor/execExpr.h | 12 ++ src/include/executor/executor.h | 6 + src/include/executor/tuptable.h | 43 +++++ src/include/nodes/plannodes.h | 20 ++ 13 files changed, 579 insertions(+), 36 deletions(-) diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index f36fbb60101..19b1bfee135 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -7003,6 +7003,7 @@ SELECT * FROM parent WHERE key = 2400; + @@ -12851,6 +12852,31 @@ dynamic_library_path = '/usr/local/lib/postgresql:$libdir' + + detoast_reuse (boolean) + + detoast_reuse configuration parameter + + + + + When a query references the same large (TOASTed) + column in several expressions of one plan node, the executor normally + fetches and decompresses the value once per reference. With this + parameter on, the planner marks such columns and the executor fetches + each value once per row, reusing the plain copy for the other + references. The functions that report the stored form of a value, + pg_column_size, + pg_column_compression and + pg_column_toast_chunk_id, keep seeing it when + applied to the column directly; a function that receives the column + as an argument and calls them internally sees the fetched value, as + it would for any other argument. The parameter is consulted when a + query is planned; setting it off restores the per-reference behavior + for plans made afterwards. The default is on. + + + debug_discard_caches (integer) diff --git a/src/backend/executor/README b/src/backend/executor/README index 54f4782f31b..8afe320ca1a 100644 --- a/src/backend/executor/README +++ b/src/backend/executor/README @@ -267,6 +267,50 @@ executed for that row. Statement triggers are fired always, regardless of whether any rows match the corresponding clauses. +Detoasting a Column Once Per Row +-------------------------------- + +A TOASTed value is normally fetched and decompressed by every function that +receives it, so a query mentioning the same large column in several +expressions of one plan node pays for it several times. The executor can +instead detoast such a column once per row and hand the same plain value to +every reference in that node. + +The value is kept beside the slot, in TupleTableSlot.tts_detoasted, for as +long as the slot holds the tuple. tts_values is never modified: it keeps the +stored datum, out of line or compressed as it was. That is the invariant the +whole mechanism rests on, and it is what makes the rest of the executor need +no changes. Every other consumer reads tts_values and is therefore +unaffected, whether it copies a slot, stores it in a tuple, a tuplestore, a +hash table or a batch file, projects the column onward, or inspects its stored +form. + +Only argument positions read the copy. An argument position is a place where +a construct consumes a value and does not return it: the arguments of a +function or operator call, the two sides of a scalar array operation, the +input of an I/O or array coercion, a field or subscript extraction, the +elements of an array construction, the sides of a row comparison and the +argument of a hash function built for a hash join or hashed grouping. A Var +anywhere else, in particular a bare targetlist entry or an input of CASE, +COALESCE, GREATEST, LEAST or NULLIF, all of which can return their input +unchanged, compiles as it always did and yields the stored datum. This is +why a detoasted copy can never become the result of an expression and never +reaches code that must see the stored form. + +Two lists implement that rule: the argument positions themselves, which are +the calls to ExecInitDetoastArg() in execExpr.c, and the functions whose +result depends on the stored representation and which must therefore receive +it, ExecFuncReadsStoredForm() in the same file. + +Lifetime. A copy belongs to the tuple the slot holds. A slot implementation +promises, through TupleTableSlotOps.resets_detoasted, to release the copies +whenever it invalidates tts_values, in its store, clear and materialize paths; +an implementation that leaves the field false keeps no copies at all and +simply detoasts per reference. + +Which columns are worth a copy is not decided here. ExecInitDetoastArg() +reads the sets on the Plan node, which nothing fills in yet. + Memory Management ----------------- diff --git a/src/backend/executor/execExpr.c b/src/backend/executor/execExpr.c index 82e846a1f4f..015c89464d5 100644 --- a/src/backend/executor/execExpr.c +++ b/src/backend/executor/execExpr.c @@ -47,12 +47,16 @@ #include "utils/acl.h" #include "utils/array.h" #include "utils/builtins.h" +#include "utils/fmgroids.h" #include "utils/jsonfuncs.h" #include "utils/jsonpath.h" #include "utils/lsyscache.h" #include "utils/typcache.h" +/* GUC */ +bool detoast_reuse = true; + typedef struct ExprSetupInfo { /* @@ -69,11 +73,15 @@ typedef struct ExprSetupInfo } ExprSetupInfo; static void ExecReadyExpr(ExprState *state); +static bool ExecPushDetoastArgStep(Expr *expr, ExprState *state, + Datum *resv, bool *resnull); +static inline void ExecInitDetoastArg(Expr *arg, ExprState *state, + Datum *resv, bool *resnull); static void ExecInitExprRec(Expr *node, ExprState *state, Datum *resv, bool *resnull); static void ExecInitFunc(ExprEvalStep *scratch, Expr *node, List *args, Oid funcid, Oid inputcollid, - ExprState *state); + ExprState *state, bool detoast_args); static void ExecInitSubPlanExpr(SubPlan *subplan, ExprState *state, Datum *resv, bool *resnull); @@ -435,7 +443,9 @@ ExecBuildProjectionInfo(List *targetList, if (isSafeVar) { - /* Fast-path: just generate an EEOP_ASSIGN_*_VAR step */ + /* + * Fast-path: just generate an EEOP_ASSIGN_*_VAR step + */ switch (variable->varno) { case INNER_VAR: @@ -889,6 +899,147 @@ ExecCheck(ExprState *state, ExprContext *econtext) return DatumGetBool(ret); } +/* + * Functions whose result depends on how the argument is stored rather than on + * its value, and which must therefore receive the stored datum. + */ +bool +ExecFuncReadsStoredForm(Oid funcid) +{ + switch (funcid) + { + case F_PG_COLUMN_SIZE: + case F_PG_COLUMN_COMPRESSION: + case F_PG_COLUMN_TOAST_CHUNK_ID: + return true; + default: + return false; + } +} + +/* + * Functions that read at most a prefix or the size of a varlena argument + * (pg_detoast_datum_slice, toast_raw_datum_size) rather than the whole value. + * Detoasting the whole value for one of these would cost more than it saves. + * length(text) is not among them: it counts characters, which needs the whole + * value in a multibyte encoding. + */ +bool +ExecFuncReadsSliceOrSize(Oid funcid) +{ + switch (funcid) + { + case F_SUBSTR_TEXT_INT4: + case F_SUBSTR_TEXT_INT4_INT4: + case F_SUBSTRING_TEXT_INT4: + case F_SUBSTRING_TEXT_INT4_INT4: + case F_SUBSTR_BYTEA_INT4: + case F_SUBSTR_BYTEA_INT4_INT4: + case F_SUBSTRING_BYTEA_INT4: + case F_SUBSTRING_BYTEA_INT4_INT4: + case F_STARTS_WITH: + case F_LEFT: + case F_RIGHT: + case F_OVERLAY_TEXT_TEXT_INT4: + case F_OVERLAY_TEXT_TEXT_INT4_INT4: + case F_OVERLAY_BYTEA_BYTEA_INT4: + case F_OVERLAY_BYTEA_BYTEA_INT4_INT4: + case F_OCTET_LENGTH_TEXT: + case F_OCTET_LENGTH_BYTEA: + case F_LENGTH_BYTEA: + return true; + default: + return false; + } +} + +/* + * Prepare evaluation of an argument whose whole value the consuming step + * reads. A Var can have a copy to hand out when its own node detoasts + * something once per row, a PARAM_EXEC parameter when any node of the plan + * does, since the copy it finds was made elsewhere; everything else, the + * common case, is ExecInitExprRec without further ado. + */ +static inline void +ExecInitDetoastArg(Expr *arg, ExprState *state, Datum *resv, bool *resnull) +{ + PlanState *parent = state->parent; + Expr *expr = arg; + Plan *plan; + + /* + * Expressions outside a plan tree, and the ones COPY compiles against a + * ModifyTableState it builds itself, have no plan to ask. + */ + if (parent == NULL) + { + ExecInitExprRec(arg, state, resv, resnull); + return; + } + plan = parent->plan; + + while (IsA(expr, RelabelType)) + expr = ((RelabelType *) expr)->arg; + if (!((IsA(expr, Var) && plan != NULL && + (plan->detoast_reuse_scan != NULL || + plan->detoast_reuse_outer != NULL || + plan->detoast_reuse_inner != NULL))) || + !ExecPushDetoastArgStep(expr, state, resv, resnull)) + ExecInitExprRec(arg, state, resv, resnull); +} + + +/* + * Out-of-line part of ExecInitDetoastArg: push the step that hands out the + * detoasted copy if the argument qualifies, and say whether it did. A plain + * Var of an attribute the node detoasts once per row becomes the + * corresponding EEOP_*_VAR_DETOAST step; a varlena PARAM_EXEC parameter becomes + * EEOP_PARAM_EXEC_DETOAST, which does the same for a parameter set from a slot + * column. Everything else, and every Var or Param in any other position, + * goes through ExecInitExprRec and sees the stored datum. Restricting the + * steps to argument positions is what keeps a detoasted copy from ever + * becoming an expression result: the constructs that return an input + * unchanged (bare Vars, RelabelType, CASE, COALESCE, GREATEST/LEAST, NULLIF) + * never get one. + */ +static bool +ExecPushDetoastArgStep(Expr *expr, ExprState *state, Datum *resv, bool *resnull) +{ + ExprEvalStep scratch = {0}; + + { + Var *var = (Var *) expr; + Plan *plan = state->parent->plan; + Bitmapset *attrs; + + if (var->varattno <= 0 || var->varreturningtype != VAR_RETURNING_DEFAULT) + return false; + switch (var->varno) + { + case INNER_VAR: + attrs = plan->detoast_reuse_inner; + scratch.opcode = EEOP_INNER_VAR_DETOAST; + break; + case OUTER_VAR: + attrs = plan->detoast_reuse_outer; + scratch.opcode = EEOP_OUTER_VAR_DETOAST; + break; + default: + attrs = plan->detoast_reuse_scan; + scratch.opcode = EEOP_SCAN_VAR_DETOAST; + break; + } + if (!bms_is_member(var->varattno, attrs)) + return false; + scratch.d.var.attnum = var->varattno - 1; + scratch.d.var.vartype = var->vartype; + } + scratch.resvalue = resv; + scratch.resnull = resnull; + ExprEvalPushStep(state, &scratch); + return true; +} + /* * Prepare a compiled expression for execution. This has to be called for * every ExprState before it can be executed. @@ -1195,7 +1346,7 @@ ExecInitExprRec(Expr *node, ExprState *state, ExecInitFunc(&scratch, node, func->args, func->funcid, func->inputcollid, - state); + state, !ExecFuncReadsStoredForm(func->funcid)); ExprEvalPushStep(state, &scratch); break; } @@ -1206,7 +1357,7 @@ ExecInitExprRec(Expr *node, ExprState *state, ExecInitFunc(&scratch, node, op->args, op->opfuncid, op->inputcollid, - state); + state, true); ExprEvalPushStep(state, &scratch); break; } @@ -1217,7 +1368,7 @@ ExecInitExprRec(Expr *node, ExprState *state, ExecInitFunc(&scratch, node, op->args, op->opfuncid, op->inputcollid, - state); + state, true); /* * Change opcode of call instruction to EEOP_DISTINCT. @@ -1237,9 +1388,14 @@ ExecInitExprRec(Expr *node, ExprState *state, { NullIfExpr *op = (NullIfExpr *) node; + /* + * NULLIF returns its first argument when the comparison says + * the values differ, so its arguments are not consumed and + * must not be handed a detoasted copy. + */ ExecInitFunc(&scratch, node, op->args, op->opfuncid, op->inputcollid, - state); + state, false); /* * If first argument is of varlena type, we'll need to ensure @@ -1329,8 +1485,8 @@ ExecInitExprRec(Expr *node, ExprState *state, if (OidIsValid(opexpr->hashfuncid)) { /* Evaluate scalar directly into left function argument */ - ExecInitExprRec(scalararg, state, - &fcinfo->args[0].value, &fcinfo->args[0].isnull); + ExecInitDetoastArg(scalararg, state, + &fcinfo->args[0].value, &fcinfo->args[0].isnull); /* * Evaluate array argument into our return value. There's @@ -1339,7 +1495,7 @@ ExecInitExprRec(Expr *node, ExprState *state, * EEOP_HASHED_SCALARARRAYOP, and will not be passed to * any other expression. */ - ExecInitExprRec(arrayarg, state, resv, resnull); + ExecInitDetoastArg(arrayarg, state, resv, resnull); /* And perform the operation */ scratch.opcode = EEOP_HASHED_SCALARARRAYOP; @@ -1354,9 +1510,9 @@ ExecInitExprRec(Expr *node, ExprState *state, else { /* Evaluate scalar directly into left function argument */ - ExecInitExprRec(scalararg, state, - &fcinfo->args[0].value, - &fcinfo->args[0].isnull); + ExecInitDetoastArg(scalararg, state, + &fcinfo->args[0].value, + &fcinfo->args[0].isnull); /* * Evaluate array argument into our return value. There's @@ -1364,7 +1520,7 @@ ExecInitExprRec(Expr *node, ExprState *state, * guaranteed to be overwritten by EEOP_SCALARARRAYOP, and * will not be passed to any other expression. */ - ExecInitExprRec(arrayarg, state, resv, resnull); + ExecInitDetoastArg(arrayarg, state, resv, resnull); /* And perform the operation */ scratch.opcode = EEOP_SCALARARRAYOP; @@ -1492,7 +1648,7 @@ ExecInitExprRec(Expr *node, ExprState *state, FieldSelect *fselect = (FieldSelect *) node; /* evaluate row/record argument into result area */ - ExecInitExprRec(fselect->arg, state, resv, resnull); + ExecInitDetoastArg(fselect->arg, state, resv, resnull); /* and extract field */ scratch.opcode = EEOP_FIELDSELECT; @@ -1618,7 +1774,7 @@ ExecInitExprRec(Expr *node, ExprState *state, FunctionCallInfo fcinfo_in; /* evaluate argument into step's result area */ - ExecInitExprRec(iocoerce->arg, state, resv, resnull); + ExecInitDetoastArg(iocoerce->arg, state, resv, resnull); /* * Prepare both output and input function calls, to be @@ -1680,7 +1836,7 @@ ExecInitExprRec(Expr *node, ExprState *state, ExprState *elemstate; /* evaluate argument into step's result area */ - ExecInitExprRec(acoerce->arg, state, resv, resnull); + ExecInitDetoastArg(acoerce->arg, state, resv, resnull); resultelemtype = get_element_type(acoerce->resulttype); if (!OidIsValid(resultelemtype)) @@ -1950,9 +2106,9 @@ ExecInitExprRec(Expr *node, ExprState *state, { Expr *e = (Expr *) lfirst(lc); - ExecInitExprRec(e, state, - &scratch.d.arrayexpr.elemvalues[elemoff], - &scratch.d.arrayexpr.elemnulls[elemoff]); + ExecInitDetoastArg(e, state, + &scratch.d.arrayexpr.elemvalues[elemoff], + &scratch.d.arrayexpr.elemnulls[elemoff]); elemoff++; } @@ -2123,10 +2279,10 @@ ExecInitExprRec(Expr *node, ExprState *state, */ /* evaluate left and right args directly into fcinfo */ - ExecInitExprRec(left_expr, state, - &fcinfo->args[0].value, &fcinfo->args[0].isnull); - ExecInitExprRec(right_expr, state, - &fcinfo->args[1].value, &fcinfo->args[1].isnull); + ExecInitDetoastArg(left_expr, state, + &fcinfo->args[0].value, &fcinfo->args[0].isnull); + ExecInitDetoastArg(right_expr, state, + &fcinfo->args[1].value, &fcinfo->args[1].isnull); scratch.opcode = EEOP_ROWCOMPARE_STEP; scratch.d.rowcompare_step.finfo = finfo; @@ -2694,7 +2850,7 @@ ExprEvalPushStep(ExprState *es, const ExprEvalStep *s) */ static void ExecInitFunc(ExprEvalStep *scratch, Expr *node, List *args, Oid funcid, - Oid inputcollid, ExprState *state) + Oid inputcollid, ExprState *state, bool detoast_args) { int nargs = list_length(args); AclResult aclresult; @@ -2769,9 +2925,14 @@ ExecInitFunc(ExprEvalStep *scratch, Expr *node, List *args, Oid funcid, } else { - ExecInitExprRec(arg, state, - &fcinfo->args[argno].value, - &fcinfo->args[argno].isnull); + if (detoast_args) + ExecInitDetoastArg(arg, state, + &fcinfo->args[argno].value, + &fcinfo->args[argno].isnull); + else + ExecInitExprRec(arg, state, + &fcinfo->args[argno].value, + &fcinfo->args[argno].isnull); } argno++; } @@ -3298,7 +3459,7 @@ ExecInitSubscriptingRef(ExprEvalStep *scratch, SubscriptingRef *sbsref, * be overwritten by the final EEOP_SBSREF_FETCH/ASSIGN step, which is * pushed last. */ - ExecInitExprRec(sbsref->refexpr, state, resv, resnull); + ExecInitDetoastArg(sbsref->refexpr, state, resv, resnull); /* * If refexpr yields NULL, and the operation should be strict, then result @@ -4375,10 +4536,10 @@ ExecBuildHash32Expr(TupleDesc desc, const TupleTableSlotOps *ops, * Build the steps to evaluate the hash function's argument, placing * the value in the 0th argument of the hash func. */ - ExecInitExprRec(expr, - state, - &fcinfo->args[0].value, - &fcinfo->args[0].isnull); + ExecInitDetoastArg(expr, + state, + &fcinfo->args[0].value, + &fcinfo->args[0].isnull); if (i == num_exprs - 1) { diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c index 397219f7a3a..97d140e6e5f 100644 --- a/src/backend/executor/execExprInterp.c +++ b/src/backend/executor/execExprInterp.c @@ -56,6 +56,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/heaptoast.h" #include "access/tupconvert.h" #include "catalog/pg_type.h" @@ -457,6 +458,32 @@ ExecReadyInterpretedExpr(ExprState *state) } +/* + * Inline part of the EEOP_*_VAR_DETOAST steps. Values that are neither + * compressed nor stored out of line, the common case for short strings, are + * handed out as they are without leaving the interpreter loop; the others go + * to ExecEvalVarDetoastSlow for the copy beside the slot. + */ +static void ExecEvalVarDetoastSlow(ExprEvalStep *op, TupleTableSlot *slot); + +static inline void +ExecEvalVarDetoastInline(ExprEvalStep *op, TupleTableSlot *slot) +{ + int attnum = op->d.var.attnum; + Datum value = slot->tts_values[attnum]; + + Assert(attnum >= 0 && attnum < slot->tts_nvalid); + if (!slot->tts_isnull[attnum] && + (VARATT_IS_COMPRESSED(DatumGetPointer(value)) || + VARATT_IS_EXTERNAL(DatumGetPointer(value)))) + ExecEvalVarDetoastSlow(op, slot); + else + { + *op->resvalue = value; + *op->resnull = slot->tts_isnull[attnum]; + } +} + /* * Evaluate expression identified by "state" in the execution context * given by "econtext". *isnull is set to the is-null flag for the result, @@ -494,6 +521,9 @@ ExecInterpExpr(ExprState *state, ExprContext *econtext, bool *isnull) &&CASE_EEOP_SCAN_VAR, &&CASE_EEOP_OLD_VAR, &&CASE_EEOP_NEW_VAR, + &&CASE_EEOP_INNER_VAR_DETOAST, + &&CASE_EEOP_OUTER_VAR_DETOAST, + &&CASE_EEOP_SCAN_VAR_DETOAST, &&CASE_EEOP_INNER_SYSVAR, &&CASE_EEOP_OUTER_SYSVAR, &&CASE_EEOP_SCAN_SYSVAR, @@ -755,6 +785,27 @@ ExecInterpExpr(ExprState *state, ExprContext *econtext, bool *isnull) EEO_NEXT(); } + EEO_CASE(EEOP_INNER_VAR_DETOAST) + { + ExecEvalVarDetoastInline(op, innerslot); + + EEO_NEXT(); + } + + EEO_CASE(EEOP_OUTER_VAR_DETOAST) + { + ExecEvalVarDetoastInline(op, outerslot); + + EEO_NEXT(); + } + + EEO_CASE(EEOP_SCAN_VAR_DETOAST) + { + ExecEvalVarDetoastInline(op, scanslot); + + EEO_NEXT(); + } + EEO_CASE(EEOP_INNER_SYSVAR) { ExecEvalSysVar(state, op, econtext, innerslot); @@ -844,6 +895,7 @@ ExecInterpExpr(ExprState *state, ExprContext *econtext, bool *isnull) EEO_NEXT(); } + EEO_CASE(EEOP_ASSIGN_OLD_VAR) { int resultnum = op->d.assign_var.resultnum; @@ -1318,6 +1370,7 @@ ExecInterpExpr(ExprState *state, ExprContext *econtext, bool *isnull) EEO_NEXT(); } + EEO_CASE(EEOP_PARAM_EXEC) { /* out of line implementation: too large */ @@ -2335,6 +2388,7 @@ CheckExprStillValid(ExprState *state, ExprContext *econtext) switch (ExecEvalStepOp(state, op)) { case EEOP_INNER_VAR: + case EEOP_INNER_VAR_DETOAST: { int attnum = op->d.var.attnum; @@ -2343,6 +2397,7 @@ CheckExprStillValid(ExprState *state, ExprContext *econtext) } case EEOP_OUTER_VAR: + case EEOP_OUTER_VAR_DETOAST: { int attnum = op->d.var.attnum; @@ -2351,6 +2406,7 @@ CheckExprStillValid(ExprState *state, ExprContext *econtext) } case EEOP_SCAN_VAR: + case EEOP_SCAN_VAR_DETOAST: { int attnum = op->d.var.attnum; @@ -5655,6 +5711,103 @@ ExecEvalWholeRowVar(ExprState *state, ExprEvalStep *op, ExprContext *econtext) *op->resnull = false; } +/* + * The detoasted copy of the slot's attnum'th value, made on first use. + * + * This detoasts unconditionally rather than rechecking, so the caller is + * responsible for the whole precondition: attr must be the slot's attnum'th + * value, that value must not be null, and it must be stored out of line or + * compressed, so that detoasting it yields something different. The + * assertions below state that contract; the callers establish it with the + * test they need anyway to take this path at all. + * + * Copies live in the slot's detoast context, which the slot implementation + * resets whenever tts_values is invalidated; tts_values itself is never + * modified. An implementation that does not promise those resets keeps no + * copies at all, and its values are detoasted once per reference as before. + */ +static Datum +slot_detoast_attr(TupleTableSlot *slot, int attnum, varlena *attr) +{ + Assert(attnum >= 0 && attnum < slot->tts_nvalid); + Assert(!slot->tts_isnull[attnum]); + Assert(PointerGetDatum(attr) == slot->tts_values[attnum]); + Assert(VARATT_IS_COMPRESSED(attr) || VARATT_IS_EXTERNAL(attr)); + + if (!slot->tts_ops->resets_detoasted) + return PointerGetDatum(detoast_attr(attr)); + + if (unlikely(slot->tts_detoast_cxt == NULL)) + slot->tts_detoast_cxt = + GenerationContextCreate(slot->tts_mcxt, + "detoasted slot values", + ALLOCSET_DEFAULT_SIZES); + if (slot->tts_detoasted == NULL) + { + slot->tts_detoasted = + MemoryContextAllocZero(slot->tts_detoast_cxt, + slot->tts_tupleDescriptor->natts * + sizeof(Datum)); +#ifdef USE_ASSERT_CHECKING + slot->tts_detoast_src = + MemoryContextAllocZero(slot->tts_detoast_cxt, + slot->tts_tupleDescriptor->natts * + sizeof(Datum)); +#endif + } + if (slot->tts_detoasted[attnum] == (Datum) 0) + { + MemoryContext oldcxt = MemoryContextSwitchTo(slot->tts_detoast_cxt); + + slot->tts_detoasted[attnum] = PointerGetDatum(detoast_attr(attr)); + MemoryContextSwitchTo(oldcxt); +#ifdef USE_ASSERT_CHECKING + slot->tts_detoast_src[attnum] = slot->tts_values[attnum]; +#endif + } + + /* + * Handing out a copy made for an earlier tuple is the one way this + * mechanism can produce a wrong answer, and it is what resets_detoasted + * exists to prevent. Check the promise where we can: the stored datum + * must still be the one the copy was made from. An address can be + * reused, so this does not prove a reset happened, but it catches a slot + * refilled without the copies being dropped, which is the mistake an + * implementation actually makes. + */ + Assert(slot->tts_detoast_src[attnum] == slot->tts_values[attnum]); + + return slot->tts_detoasted[attnum]; +} + +/* + * Out-of-line part of the EEOP_*_VAR_DETOAST steps. The value really is + * stored out of line or compressed, so take the copy beside the slot, making + * it if this is the first reference to the column while the slot holds this + * tuple. + */ +static void +ExecEvalVarDetoastSlow(ExprEvalStep *op, TupleTableSlot *slot) +{ + int attnum = op->d.var.attnum; + varlena *attr = (varlena *) DatumGetPointer(slot->tts_values[attnum]); + + *op->resvalue = slot_detoast_attr(slot, attnum, attr); + *op->resnull = false; +} + +/* + * The whole of an EEOP_*_VAR_DETOAST step, for JIT-compiled expressions, + * which call this rather than having the fast path emitted inline. econtext + * is unused; the signature is the one build_EvalXFunc() expects. + */ +void +ExecEvalVarDetoast(ExprState *state, ExprEvalStep *op, ExprContext *econtext, + TupleTableSlot *slot) +{ + ExecEvalVarDetoastInline(op, slot); +} + void ExecEvalSysVar(ExprState *state, ExprEvalStep *op, ExprContext *econtext, TupleTableSlot *slot) diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c index b8e8f52c64c..5ed5ecc5f23 100644 --- a/src/backend/executor/execTuples.c +++ b/src/backend/executor/execTuples.c @@ -184,6 +184,14 @@ tts_virtual_materialize(TupleTableSlot *slot) if (TTS_SHOULDFREE(slot)) return; + /* + * Detoasted copies carried in from another slot (see + * ExecEvalAssignVarDetoast) point into that slot, which a materialized + * slot may outlive. Drop them along with the slot's own copies; they are + * recreated on demand. + */ + ExecSlotResetDetoast(slot); + /* compute size of memory required */ for (int natt = 0; natt < desc->natts; natt++) { @@ -401,6 +409,9 @@ tts_heap_materialize(TupleTableSlot *slot) HeapTupleTableSlot *hslot = (HeapTupleTableSlot *) slot; MemoryContext oldContext; + /* see tts_virtual_materialize: carried detoasted copies must go */ + ExecSlotResetDetoast(slot); + Assert(!TTS_EMPTY(slot)); /* If slot has its tuple already materialized, nothing to do. */ @@ -488,6 +499,9 @@ tts_heap_store_tuple(TupleTableSlot *slot, HeapTuple tuple, bool shouldFree) { HeapTupleTableSlot *hslot = (HeapTupleTableSlot *) slot; + /* the slot moves on to another tuple; drop its detoasted copies */ + ExecSlotResetDetoast(slot); + tts_heap_clear(slot); slot->tts_nvalid = 0; @@ -589,6 +603,9 @@ tts_minimal_materialize(TupleTableSlot *slot) MinimalTupleTableSlot *mslot = (MinimalTupleTableSlot *) slot; MemoryContext oldContext; + /* see tts_virtual_materialize: carried detoasted copies must go */ + ExecSlotResetDetoast(slot); + Assert(!TTS_EMPTY(slot)); /* If slot has its tuple already materialized, nothing to do. */ @@ -683,6 +700,9 @@ tts_minimal_store_tuple(TupleTableSlot *slot, MinimalTuple mtup, bool shouldFree { MinimalTupleTableSlot *mslot = (MinimalTupleTableSlot *) slot; + /* the slot moves on to another tuple; drop its detoasted copies */ + ExecSlotResetDetoast(slot); + tts_minimal_clear(slot); Assert(!TTS_SHOULDFREE(slot)); @@ -806,6 +826,9 @@ tts_buffer_heap_materialize(TupleTableSlot *slot) BufferHeapTupleTableSlot *bslot = (BufferHeapTupleTableSlot *) slot; MemoryContext oldContext; + /* see tts_virtual_materialize: carried detoasted copies must go */ + ExecSlotResetDetoast(slot); + Assert(!TTS_EMPTY(slot)); /* If slot has its tuple already materialized, nothing to do. */ @@ -955,6 +978,8 @@ tts_buffer_heap_store_tuple(TupleTableSlot *slot, HeapTuple tuple, slot->tts_flags &= ~TTS_FLAG_SHOULDFREE; } + /* every refill path, including copyslot, invalidates tts_values */ + ExecSlotResetDetoast(slot); slot->tts_flags &= ~TTS_FLAG_EMPTY; slot->tts_nvalid = 0; bslot->base.tuple = tuple; @@ -1291,7 +1316,8 @@ const TupleTableSlotOps TTSOpsVirtual = { .get_heap_tuple = NULL, .get_minimal_tuple = NULL, .copy_heap_tuple = tts_virtual_copy_heap_tuple, - .copy_minimal_tuple = tts_virtual_copy_minimal_tuple + .copy_minimal_tuple = tts_virtual_copy_minimal_tuple, + .resets_detoasted = true, }; const TupleTableSlotOps TTSOpsHeapTuple = { @@ -1309,7 +1335,8 @@ const TupleTableSlotOps TTSOpsHeapTuple = { /* A heap tuple table slot can not "own" a minimal tuple. */ .get_minimal_tuple = NULL, .copy_heap_tuple = tts_heap_copy_heap_tuple, - .copy_minimal_tuple = tts_heap_copy_minimal_tuple + .copy_minimal_tuple = tts_heap_copy_minimal_tuple, + .resets_detoasted = true, }; const TupleTableSlotOps TTSOpsMinimalTuple = { @@ -1327,7 +1354,8 @@ const TupleTableSlotOps TTSOpsMinimalTuple = { .get_heap_tuple = NULL, .get_minimal_tuple = tts_minimal_get_minimal_tuple, .copy_heap_tuple = tts_minimal_copy_heap_tuple, - .copy_minimal_tuple = tts_minimal_copy_minimal_tuple + .copy_minimal_tuple = tts_minimal_copy_minimal_tuple, + .resets_detoasted = true, }; const TupleTableSlotOps TTSOpsBufferHeapTuple = { @@ -1345,7 +1373,8 @@ const TupleTableSlotOps TTSOpsBufferHeapTuple = { /* A buffer heap tuple table slot can not "own" a minimal tuple. */ .get_minimal_tuple = NULL, .copy_heap_tuple = tts_buffer_heap_copy_heap_tuple, - .copy_minimal_tuple = tts_buffer_heap_copy_minimal_tuple + .copy_minimal_tuple = tts_buffer_heap_copy_minimal_tuple, + .resets_detoasted = true, }; @@ -1476,6 +1505,11 @@ ExecResetTupleTable(List *tupleTable, /* tuple table */ /* Always release resources and reset the slot to empty */ ExecClearTuple(slot); slot->tts_ops->release(slot); + if (slot->tts_detoast_cxt) + { + MemoryContextDelete(slot->tts_detoast_cxt); + slot->tts_detoast_cxt = NULL; + } if (slot->tts_tupleDescriptor) { ReleaseTupleDesc(slot->tts_tupleDescriptor); @@ -1533,6 +1567,8 @@ ExecDropSingleTupleTableSlot(TupleTableSlot *slot) Assert(IsA(slot, TupleTableSlot)); ExecClearTuple(slot); slot->tts_ops->release(slot); + if (slot->tts_detoast_cxt) + MemoryContextDelete(slot->tts_detoast_cxt); if (slot->tts_tupleDescriptor) ReleaseTupleDesc(slot->tts_tupleDescriptor); if (!TTS_FIXED(slot)) @@ -1819,6 +1855,19 @@ ExecForceStoreMinimalTuple(MinimalTuple mtup, } } +/* + * Out-of-line part of ExecSlotResetDetoast(); see tuptable.h. + */ +void +ExecResetSlotDetoastContext(TupleTableSlot *slot) +{ + MemoryContextReset(slot->tts_detoast_cxt); + slot->tts_detoasted = NULL; +#ifdef USE_ASSERT_CHECKING + slot->tts_detoast_src = NULL; +#endif +} + /* -------------------------------- * ExecStoreVirtualTuple * Mark a slot as containing a virtual tuple. diff --git a/src/backend/jit/llvm/llvmjit_expr.c b/src/backend/jit/llvm/llvmjit_expr.c index fc80d3e55fe..288f8b7f2fc 100644 --- a/src/backend/jit/llvm/llvmjit_expr.c +++ b/src/backend/jit/llvm/llvmjit_expr.c @@ -488,6 +488,26 @@ llvm_compile_expr(ExprState *state) break; } + case EEOP_INNER_VAR_DETOAST: + case EEOP_OUTER_VAR_DETOAST: + case EEOP_SCAN_VAR_DETOAST: + { + LLVMValueRef v_slot; + + if (opcode == EEOP_INNER_VAR_DETOAST) + v_slot = v_innerslot; + else if (opcode == EEOP_OUTER_VAR_DETOAST) + v_slot = v_outerslot; + else + v_slot = v_scanslot; + + build_EvalXFunc(b, mod, "ExecEvalVarDetoast", + v_state, op, v_econtext, v_slot); + + LLVMBuildBr(b, opblocks[opno + 1]); + break; + } + case EEOP_INNER_SYSVAR: case EEOP_OUTER_SYSVAR: case EEOP_SCAN_SYSVAR: diff --git a/src/backend/jit/llvm/llvmjit_types.c b/src/backend/jit/llvm/llvmjit_types.c index c8a1f841293..034f0193e98 100644 --- a/src/backend/jit/llvm/llvmjit_types.c +++ b/src/backend/jit/llvm/llvmjit_types.c @@ -169,6 +169,7 @@ void *referenced_functions[] = ExecEvalHashedScalarArrayOp, ExecEvalSubPlan, ExecEvalSysVar, + ExecEvalVarDetoast, ExecEvalWholeRowVar, ExecEvalXmlExpr, ExecEvalJsonConstructor, diff --git a/src/backend/utils/misc/guc_parameters.dat b/src/backend/utils/misc/guc_parameters.dat index c57441f7d98..f8e924bdf05 100644 --- a/src/backend/utils/misc/guc_parameters.dat +++ b/src/backend/utils/misc/guc_parameters.dat @@ -818,6 +818,13 @@ check_hook => 'check_default_with_oids', }, +{ name => 'detoast_reuse', type => 'bool', context => 'PGC_USERSET', group => 'DEVELOPER_OPTIONS', + short_desc => 'Allow an expression to detoast a column once and reuse the value.', + flags => 'GUC_NOT_IN_SAMPLE', + variable => 'detoast_reuse', + boot_val => 'true', +}, + { name => 'dynamic_library_path', type => 'string', context => 'PGC_SUSET', group => 'CLIENT_CONN_OTHER', short_desc => 'Sets the path for dynamically loadable modules.', long_desc => 'If a dynamically loadable module needs to be opened and the specified name does not have a directory component (i.e., the name does not contain a slash), the system will search this path for the specified file.', diff --git a/src/backend/utils/misc/guc_tables.c b/src/backend/utils/misc/guc_tables.c index 342aaeef59a..0f7e3483ae8 100644 --- a/src/backend/utils/misc/guc_tables.c +++ b/src/backend/utils/misc/guc_tables.c @@ -48,6 +48,7 @@ #include "commands/trigger.h" #include "commands/user.h" #include "commands/vacuum.h" +#include "executor/executor.h" #include "common/file_utils.h" #include "common/scram-common.h" #include "jit/jit.h" diff --git a/src/include/executor/execExpr.h b/src/include/executor/execExpr.h index c61b3d624d5..469fd4be4ae 100644 --- a/src/include/executor/execExpr.h +++ b/src/include/executor/execExpr.h @@ -85,6 +85,16 @@ typedef enum ExprEvalOp EEOP_OLD_VAR, EEOP_NEW_VAR, + /* + * Same, but detoasting the value once per row and keeping the copy beside + * the slot. There are no OLD/NEW variants: the planner marks attributes + * of a node's scan, outer and inner inputs only, so a Var of the old or + * new tuple is never compiled to one of these. + */ + EEOP_INNER_VAR_DETOAST, + EEOP_OUTER_VAR_DETOAST, + EEOP_SCAN_VAR_DETOAST, + /* compute system Var value */ EEOP_INNER_SYSVAR, EEOP_OUTER_SYSVAR, @@ -906,6 +916,8 @@ extern void ExecEvalWholeRowVar(ExprState *state, ExprEvalStep *op, ExprContext *econtext); extern void ExecEvalSysVar(ExprState *state, ExprEvalStep *op, ExprContext *econtext, TupleTableSlot *slot); +extern void ExecEvalVarDetoast(ExprState *state, ExprEvalStep *op, + ExprContext *econtext, TupleTableSlot *slot); extern void ExecAggInitGroup(AggState *aggstate, AggStatePerTrans pertrans, AggStatePerGroup pergroup, ExprContext *aggcontext); diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 23a09a70aa2..f073e54a4e0 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -91,6 +91,12 @@ extern PGDLLIMPORT ExecutorFinish_hook_type ExecutorFinish_hook; typedef void (*ExecutorEnd_hook_type) (QueryDesc *queryDesc); extern PGDLLIMPORT ExecutorEnd_hook_type ExecutorEnd_hook; +/* GUC parameter */ +extern PGDLLIMPORT bool detoast_reuse; + +extern bool ExecFuncReadsStoredForm(Oid funcid); +extern bool ExecFuncReadsSliceOrSize(Oid funcid); + /* Hook for plugins to get control in ExecCheckPermissions() */ typedef bool (*ExecutorCheckPerms_hook_type) (List *rangeTable, List *rtePermInfos, diff --git a/src/include/executor/tuptable.h b/src/include/executor/tuptable.h index 3db6c9c9bd0..f0b72aeff76 100644 --- a/src/include/executor/tuptable.h +++ b/src/include/executor/tuptable.h @@ -139,6 +139,20 @@ typedef struct TupleTableSlot * TTS_FLAG_OBEYS_NOT_NULL_CONSTRAINTS */ MemoryContext tts_mcxt; /* slot itself is in this context */ + MemoryContext tts_detoast_cxt; /* holds tts_detoasted and the copies it + * points to; created on demand, reset + * whenever tts_values is invalidated */ + Datum *tts_detoasted; /* per attribute, a detoasted copy of the + * tts_values entry, or 0; NULL until the + * first copy of the current tuple is made. + * tts_values itself always keeps the stored + * datum. */ +#ifdef USE_ASSERT_CHECKING + Datum *tts_detoast_src; /* the tts_values entry each copy was made + * from, so that a copy outliving its + * tuple can be caught; see + * slot_detoast_attr() */ +#endif ItemPointerData tts_tid; /* stored tuple's tid */ Oid tts_tableOid; /* table oid of tuple */ } TupleTableSlot; @@ -239,6 +253,20 @@ struct TupleTableSlotOps * with the minimal tuple without the need for an additional allocation. */ MinimalTuple (*copy_minimal_tuple) (TupleTableSlot *slot, Size extra); + + /* + * Does this implementation call ExecSlotResetDetoast() wherever it + * invalidates tts_values by other means than clear(), that is in every + * function that stores a new tuple, copies another slot in, or + * materializes the contents? Only then may the executor keep detoasted + * copies of this slot's values beside it (tts_detoasted); otherwise a + * copy made for one tuple could be handed out for the next. + * + * Leaving this false, as an implementation that does not know about the + * field does, only costs the optimization. See the in-tree + * implementations for what setting it entails. + */ + bool resets_detoasted; }; /* @@ -355,6 +383,7 @@ extern TupleTableSlot *ExecStoreMinimalTuple(MinimalTuple mtup, extern void ExecForceStoreMinimalTuple(MinimalTuple mtup, TupleTableSlot *slot, bool shouldFree); extern TupleTableSlot *ExecStoreVirtualTuple(TupleTableSlot *slot); +extern void ExecResetSlotDetoastContext(TupleTableSlot *slot); extern TupleTableSlot *ExecStoreAllNullTuple(TupleTableSlot *slot); extern void ExecStoreHeapTupleDatum(Datum data, TupleTableSlot *slot); extern HeapTuple ExecFetchSlotHeapTuple(TupleTableSlot *slot, bool materialize, bool *shouldFree); @@ -469,12 +498,26 @@ slot_is_current_xact_tuple(TupleTableSlot *slot) return slot->tts_ops->is_current_xact_tuple(slot); } +/* + * Release the detoasted copies made by EEOP_*_VAR_DETOAST steps or carried in + * by EEOP_ASSIGN_*_VAR_DETOAST. A slot implementation calls this whenever the + * slot's tts_values are about to be invalidated, before any pointer into that + * memory could be looked at again; see resets_detoasted above. + */ +static inline void +ExecSlotResetDetoast(TupleTableSlot *slot) +{ + if (unlikely(slot->tts_detoast_cxt != NULL)) + ExecResetSlotDetoastContext(slot); +} + /* * ExecClearTuple - clear the slot's contents */ static inline TupleTableSlot * ExecClearTuple(TupleTableSlot *slot) { + ExecSlotResetDetoast(slot); slot->tts_ops->clear(slot); return slot; diff --git a/src/include/nodes/plannodes.h b/src/include/nodes/plannodes.h index 09a1ec73180..b2f54d3a93f 100644 --- a/src/include/nodes/plannodes.h +++ b/src/include/nodes/plannodes.h @@ -83,6 +83,15 @@ typedef struct PlannedStmt /* do I set the command result tag? */ bool canSetTag; + /* + * Was this plan built with detoasting a column once per row enabled? The + * per-node decisions are in the Plan nodes; expression initialization + * consults this where there is no node to ask, namely for the parameters + * a subplan or a nestloop inner plan receives, whose copy was made + * elsewhere (see ExecInitDetoastArg). + */ + bool detoastReuse; + /* redo plan when TransactionXmin changes? */ bool transientPlan; @@ -226,6 +235,17 @@ typedef struct Plan /* engage asynchronous-capable logic? */ bool async_capable; + /* + * Attributes of the scan tuple, the outer input and the inner input that + * this node's expressions may detoast once per row, keeping the copy + * beside the slot (see set_plan_detoast_reuse in setrefs.c). The + * executor compiles argument positions reading them to EEOP_*_VAR_DETOAST + * steps. + */ + Bitmapset *detoast_reuse_scan; + Bitmapset *detoast_reuse_outer; + Bitmapset *detoast_reuse_inner; + /* * Common structural data for all Plan types. */ base-commit: b69356cd789fe963447177a0077c12c6c322c203 -- 2.55.0