From ba56242ad3f196ed302dfe057ac49bf360ad013b Mon Sep 17 00:00:00 2001 From: Tatsuo Ishii Date: Sun, 30 Aug 2026 07:20:31 +0900 Subject: [PATCH v51 6/9] Row pattern recognition patch (docs). --- doc/src/sgml/advanced.sgml | 130 ++ doc/src/sgml/func/func-window.sgml | 127 ++ doc/src/sgml/perform.sgml | 82 ++ doc/src/sgml/ref/select.sgml | 135 +- src/backend/executor/README.rpr | 1890 ++++++++++++++++++++++++++++ 5 files changed, 2362 insertions(+), 2 deletions(-) create mode 100644 src/backend/executor/README.rpr diff --git a/doc/src/sgml/advanced.sgml b/doc/src/sgml/advanced.sgml index 3286c2cf0b2..b0f929266ff 100644 --- a/doc/src/sgml/advanced.sgml +++ b/doc/src/sgml/advanced.sgml @@ -552,6 +552,136 @@ WHERE pos < 3; two rows for each department). + + Row Pattern Common Syntax can be used to perform Row Pattern Recognition + in a query. The Row Pattern Common Syntax includes four sub clauses, + which must be written in this order: + AFTER MATCH SKIP, INITIAL + or SEEK, PATTERN + and DEFINE. The first two are optional, and the + complete example at the end of this section uses both. + DEFINE defines row pattern variables along with an + expression. The expression must be a logical expression, which means + it must return TRUE, FALSE + or NULL. The expression may comprise column references + and non-volatile functions. Window functions, aggregate functions, + set-returning functions and subqueries are not allowed. An example + of DEFINE is as follows. + + +DEFINE + LOWPRICE AS price <= 100, + UP AS price > PREV(price), + DOWN AS price < PREV(price) + + + Note that PREV returns the price + column in the previous row if it's called in a context of row pattern + recognition. Thus in the second line the row pattern variable "UP" + is TRUE when the price column in the current row is + greater than the price column in the previous row. Likewise, "DOWN" + is TRUE when the + price column in the current row is lower than + the price column in the previous row. + + + Once DEFINE exists, PATTERN can be + used. PATTERN defines a sequence of rows that satisfies + conditions defined in the DEFINE clause. For example + the following PATTERN defines a sequence of rows starting + with a row satisfying "LOWPRICE", then one or more rows satisfying + "UP" and finally one or more rows satisfying "DOWN". Pattern variables can + be followed by quantifiers: "+" means one or more matches, "*" means zero + or more matches, "?" means zero or one match, "{n}" (n > 0) means exactly + n matches, "{n,}" (n >= 0) means at least n matches, "{,m}" (m > 0) means + at most m matches, and "{n,m}" (0 <= n <= m, 0 < m) means between n and m + matches. Patterns can be grouped using parentheses and combined using + alternation (the vertical bar "|" for OR). For example, "(UP DOWN)+" + matches one or more repetitions of UP followed by DOWN. If a sequence of + rows which satisfies the PATTERN is found, in the starting row all columns + or functions are shown in the target list. Note that aggregations only + look into the matched rows, rather than the whole frame. On the second or + subsequent rows the window functions that read the frame, such + as first_value(), are shown as NULL, while functions + that do not depend on the frame, such + as row_number(), are unaffected. Aggregates on + non-starting rows return their initial value: for example, + count() returns 0 and sum() + returns NULL. Rows that do not match the PATTERN behave the same way. + Example of a SELECT using + the DEFINE and PATTERN clause is as + follows. + + +SELECT company, tdate, price, + first_value(price) OVER w, + max(price) OVER w, + count(price) OVER w +FROM stock + WINDOW w AS ( + PARTITION BY company + ORDER BY tdate + ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING + AFTER MATCH SKIP PAST LAST ROW + INITIAL + PATTERN (LOWPRICE UP+ DOWN+) + DEFINE + LOWPRICE AS price <= 100, + UP AS price > PREV(price), + DOWN AS price < PREV(price) +); + + + company | tdate | price | first_value | max | count +----------+------------+-------+-------------+-----+------- + company1 | 2023-07-01 | 100 | 100 | 200 | 4 + company1 | 2023-07-02 | 200 | | | 0 + company1 | 2023-07-03 | 150 | | | 0 + company1 | 2023-07-04 | 140 | | | 0 + company1 | 2023-07-05 | 150 | | | 0 + company1 | 2023-07-06 | 90 | 90 | 130 | 4 + company1 | 2023-07-07 | 110 | | | 0 + company1 | 2023-07-08 | 130 | | | 0 + company1 | 2023-07-09 | 120 | | | 0 + company1 | 2023-07-10 | 130 | | | 0 +(10 rows) + + + + + Row Pattern Recognition internally uses a nondeterministic finite + automaton (NFA) to match patterns. For patterns with unbounded + quantifiers (e.g., A+ or (A B)+), + the NFA may need to track many active matching contexts simultaneously, + which could potentially lead to O(n2) + complexity as the number of rows increases. + + + + Before execution, PostgreSQL automatically + optimizes patterns to simplify their structure. This includes flattening + nested sequences and alternations, merging consecutive identical variables + (e.g., A{2,3} A{1,2} becomes A{3,5}), + removing duplicate alternatives + (e.g., (A | B | A) becomes (A | B)), + and simplifying nested quantifiers + (e.g., (A*)* becomes A*). + These optimizations reduce pattern complexity and also decrease + nesting depth, making the 254-level depth limit rarely encountered. + They are applied transparently and can be observed + in EXPLAIN output. + + + + To mitigate the O(n2) complexity described + above, PostgreSQL also employs + a context absorption optimization. When a pattern starts with a greedy + unbounded element, newer matching contexts cannot produce longer matches + than older contexts. By detecting and eliminating these redundant + contexts, the matching complexity is reduced from + O(n2) to O(n) for many common patterns. + + When a query involves multiple window functions, it is possible to write out each one with a separate OVER clause, but this is diff --git a/doc/src/sgml/func/func-window.sgml b/doc/src/sgml/func/func-window.sgml index bb41387f873..1079b6abb6e 100644 --- a/doc/src/sgml/func/func-window.sgml +++ b/doc/src/sgml/func/func-window.sgml @@ -278,6 +278,133 @@ IGNORE NULLS nth_value. + + Row Pattern Recognition navigation functions are listed in + . These functions + can be used to describe the DEFINE clause of Row Pattern Recognition. + The names PREV, NEXT, + FIRST, and LAST are + recognized as navigation functions only in an unqualified call; a + schema-qualified call is resolved as an ordinary function instead. + + + + Row Pattern Navigation Functions + + + + + Function + + + Description + + + + + + + + + prev + + prev ( value anyelement [, offset bigint ] ) + anyelement + + + Returns value evaluated at the row that is + offset rows before the current row within + the partition; + returns NULL if the target row is outside the partition. + offset defaults to 1 if omitted. + offset must be a non-negative integer; + an offset of 0 refers to the current row itself. + offset must not be NULL. + Can only be used in a DEFINE clause. + + + + + + + next + + next ( value anyelement [, offset bigint ] ) + anyelement + + + Returns value evaluated at the row that is + offset rows after the current row within + the partition; + returns NULL if the target row is outside the partition. + offset defaults to 1 if omitted. + offset must be a non-negative integer; + an offset of 0 refers to the current row itself. + offset must not be NULL. + Can only be used in a DEFINE clause. + + + + + + + first + + first ( value anyelement [, offset bigint ] ) + anyelement + + + Returns value evaluated at the row that is + offset rows after the match start row; + returns NULL if the target row is beyond the current row. + offset defaults to 0 if omitted, referring to the + match start row itself. + offset must be a non-negative integer. + offset must not be NULL. + Can only be used in a DEFINE clause. + + + + + + + last + + last ( value anyelement [, offset bigint ] ) + anyelement + + + Returns value evaluated at the row that is + offset rows before the current row within + the match; + returns NULL if the target row is before the match start row. + offset defaults to 0 if omitted, referring to the + current row itself. + offset must be a non-negative integer. + offset must not be NULL. + Can only be used in a DEFINE clause. + + + + + +
+ + + PREV and NEXT may wrap + FIRST or LAST for compound + navigation. For example, + PREV(FIRST(val, 2), 3) fetches the value at + 3 rows before the row that is 2 rows after the match start. + The reverse nesting (FIRST/LAST + wrapping PREV/NEXT) is not + permitted. Same-category nesting (e.g., + PREV inside PREV) is also + prohibited. + The offset argument must be a run-time constant: + it cannot reference columns or contain a navigation operation. + + The SQL standard defines a FROM FIRST or FROM LAST diff --git a/doc/src/sgml/perform.sgml b/doc/src/sgml/perform.sgml index ea8da01b779..7d0bc9b8fbe 100644 --- a/doc/src/sgml/perform.sgml +++ b/doc/src/sgml/perform.sgml @@ -702,6 +702,88 @@ FROM tenk1 t1 WHERE t1.ten = (SELECT (random() * 10)::integer); happen without the sub-SELECT construct. + + When examining query plans for Row Pattern Recognition with + EXPLAIN, the Pattern line may + carry absorption markers. They report an analysis the planner has + already made, not an opportunity left open. A number sign + # marks the comparison point of an absorption, which + is always an element carrying an unbounded quantifier, and a tilde + ~ marks an element that lies within the absorbable + region. Neither marker requires an alternation to be present. + For example, using the stock + table from , a query that looks for + repeated up-then-down movements reports its pattern as + (up~ down~)+#: + + +EXPLAIN (COSTS OFF) +SELECT company, tdate, price, count(price) OVER w +FROM stock +WINDOW w AS ( + PARTITION BY company + ORDER BY tdate + ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING + AFTER MATCH SKIP PAST LAST ROW + INITIAL + PATTERN ((UP DOWN)+) + DEFINE UP AS price > PREV(price), + DOWN AS price < PREV(price) +); + + QUERY PLAN +-------------------------------------------------------------------&zwsp;------------------------------------ + WindowAgg + Window: w AS (PARTITION BY company ORDER BY tdate ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING) + Pattern: (up~ down~)+# + Nav Mark Lookback: 1 + -> Sort + Sort Key: company, tdate + -> Seq Scan on stock + + + Here the tildes mark up and + down as lying within the absorbable region, while + the trailing number sign marks the repeated group as the comparison + point. + + + + The Nav Mark Lookback line reports the row-retention + bound the executor computed for the backward navigations used in + DEFINE (PREV and + LAST); a Nav Mark Lookahead line + appears when DEFINE navigates forward + (FIRST). Each reports a row count when the + navigation offsets are constant; the lookahead is measured from the + match start, so a compound navigation that reaches back past it, such + as PREV(FIRST(val, 1), 2), reports a negative one. + Otherwise Nav Mark Lookback reports + runtime for an offset that is not known until + execution, or retain all when no bound could be + computed and every row of the frame has to be kept; and + Nav Mark Lookahead reports runtime + or infinite. + + + + Under ANALYZE, further lines report counters for the + matcher's internal states and contexts. Those are diagnostics for the + implementation rather than a stable interface; their names and contents + may change. + + + + Neither the AFTER MATCH SKIP nor the + INITIAL subclause is printed. Nothing is lost for + INITIAL, which is the only mode currently supported, + but AFTER MATCH SKIP still shows through: absorption + is analyzed only under SKIP PAST LAST ROW, so the + same pattern prints with the # and + ~ markers in that mode and without them in the + others. + + diff --git a/doc/src/sgml/ref/select.sgml b/doc/src/sgml/ref/select.sgml index e7072691669..0a03957fd23 100644 --- a/doc/src/sgml/ref/select.sgml +++ b/doc/src/sgml/ref/select.sgml @@ -1014,8 +1014,8 @@ WINDOW window_name AS ( frame_clause can be one of -{ RANGE | ROWS | GROUPS } frame_start [ frame_exclusion ] -{ RANGE | ROWS | GROUPS } BETWEEN frame_start AND frame_end [ frame_exclusion ] +{ RANGE | ROWS | GROUPS } frame_start [ frame_exclusion ] [ row_pattern_common_syntax ] +{ RANGE | ROWS | GROUPS } BETWEEN frame_start AND frame_end [ frame_exclusion ] [ row_pattern_common_syntax ] where frame_start @@ -1122,6 +1122,112 @@ EXCLUDE NO OTHERS a given peer group will be in the frame or excluded from it. + + The + optional row_pattern_common_syntax + defines the Row Pattern Recognition condition for + this + window. row_pattern_common_syntax + includes the following subclauses. + + +[ { AFTER MATCH SKIP PAST LAST ROW | AFTER MATCH SKIP TO NEXT ROW } ] +[ INITIAL | SEEK ] +PATTERN ( pattern_variable_name [ quantifier ] [ ... ] ) +DEFINE definition_variable_name AS expression [, ...] + + AFTER MATCH SKIP PAST LAST ROW or AFTER MATCH + SKIP TO NEXT ROW controls how to proceed to the next row position + after a match is found. With AFTER MATCH SKIP PAST LAST + ROW (the default) the next row position is next to the last row of + the previous match. On the other hand, with AFTER MATCH SKIP TO NEXT + ROW the next row position is next to the first row of the previous + match. INITIAL or SEEK specifies from + which row in the frame pattern matching begins. + If INITIAL is specified, the match must start + from the first row in the frame. If SEEK is specified, + the set of matching rows does not necessarily start from the first row. The + default is INITIAL. Currently + only INITIAL is supported. DEFINE + defines definition variables along with a boolean + expression. PATTERN defines a sequence of rows that + satisfies certain conditions using variables defined + in the DEFINE clause (an empty PATTERN() + is not accepted by the syntax). Each pattern variable can be followed + by a quantifier to specify how many times it should match: + * (zero or more), + + (one or more), + ? (zero or one), + {n} (exactly n times, n > 0), + {n,} (at least n times, n >= 0), + {,m} (at most m times, m > 0), or + {n,m} + (between n and m times, 0 <= n <= m, 0 < m). + Reluctant quantifiers (e.g., *?, +?, + ??, {n,m}?) + are supported. + The exclusion ({- and -}) is not + accepted by the syntax, and the permutation + (PERMUTE) is rejected as an unsupported feature. + PERMUTE is recognized wherever + a ( follows it, so a pattern variable of that name has + to be written "permute" in that position. + Patterns can be grouped using parentheses, and alternation (OR) can be + expressed using the vertical bar |. + For example, (A B)+ matches one or more repetitions + of the sequence A followed by B, and A | B matches + either A or B. + If a pattern variable is not defined in + the DEFINE clause, it is not automatically added + to the DEFINE clause. Instead, the executor evaluates + the variable as TRUE at execution time, behaving as if + the following definition existed. + + +variable_name AS TRUE + + + Conversely, variables defined in the DEFINE clause + but not used in the PATTERN clause are rejected + with an error. The DEFINE clause itself is not + optional: leaving it out is a syntax error, even where every pattern + variable would evaluate as TRUE. + + + + Row pattern recognition constrains the frame clause it is attached to. + The frame must be written in ROWS mode + as BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING, or + as BETWEEN CURRENT ROW + AND offset FOLLOWING with + a positive offset. + RANGE and GROUPS mode, any other + frame start or end, and + any frame_exclusion option other + than EXCLUDE NO OTHERS are rejected with an error. + + + + Note that the maximum number of unique pattern variables + used in the PATTERN clause is 240. + If this limit is exceeded, an error will be raised. + Additionally, the maximum nesting depth of pattern groups + (parentheses) is 254 levels. An alternation costs a level of its own, + so a pattern that nests alternations reaches the limit at about half + as many parentheses. + However, pattern optimizations such as flattening nested sequences + and simplifying nested quantifiers may reduce the effective depth, + so this limit is rarely reached in practice. + + + + Row pattern recognition cannot appear anywhere in a common table + expression that belongs to a WITH RECURSIVE clause; + such a query is rejected with an error. The same restriction reaches + CREATE RECURSIVE VIEW, which is rewritten + to WITH RECURSIVE. + + The purpose of a WINDOW clause is to specify the behavior of window functions appearing in the query's @@ -2235,6 +2341,31 @@ SELECT 2+2; + + Row Pattern Recognition + + + PostgreSQL supports row pattern recognition + both in a WINDOW clause entry and in an + inline OVER ( ... ) specification. The SQL standard + defines more subclauses: MEASURES and + SUBSET. They are not currently supported + in PostgreSQL. Also in the standard there are + more variations in the AFTER MATCH clause. + + + + The largest difference is in DEFINE: a column + reference qualified by a pattern variable, such + as A.price, is not implemented and is rejected as an + unsupported feature, even though the standard writes + its DEFINE examples that way. An unqualified column + reference in a DEFINE expression therefore always + reads the row being tested, and other rows of the match are reachable + only through the navigation operations. + + + Nonstandard Clauses diff --git a/src/backend/executor/README.rpr b/src/backend/executor/README.rpr new file mode 100644 index 00000000000..05bbcd76240 --- /dev/null +++ b/src/backend/executor/README.rpr @@ -0,0 +1,1890 @@ +============================================================================ + PostgreSQL Row Pattern Recognition: Flat-Array Stream NFA Guide +============================================================================ + + This README's target audience is developers with a basic + understanding of the PostgreSQL executor and planner architecture. + Also it would be better for them to understand the specification of + the row pattern recognition in the SQL standard [1][2]. If you do + not have access to the SQL standard, Oracle's manual or Trino's + manual can be alternatives for them. + + This README's scope is the entire process from PATTERN/DEFINE clause + parsing to NFA runtime execution. + + Related code: + - src/backend/parser/parse_rpr.c (parser phase) + - src/backend/optimizer/plan/rpr.c (optimizer phase) + - src/backend/executor/nodeWindowAgg.c (executor phase, window agg) + - src/backend/executor/execRPR.c (executor phase, NFA engine) + - src/include/executor/execRPR.h (NFA public API) + - src/include/nodes/plannodes.h (plan node definitions) + - src/include/nodes/execnodes.h (execution state definitions) + - src/include/optimizer/rpr.h (types and constants) + - src/backend/optimizer/plan/createplan.c (match_start dependency metadata) + +============================================================================ + +What is a Flat-Array Stream NFA? + + The NFA (Nondeterministic Finite Automaton) in this implementation + is not a traditional state-transition graph but a flat array of + fixed-size 16-byte elements. At runtime, it processes the row stream + in a forward-only manner, expanding epsilon transitions eagerly + without backtracking. + + - Flat-Array: Pattern compiled into a flat array, + not a graph (Chapter IV) + - Stream: Rows consumed sequentially in one direction, + never revisited (Chapter XII) + - NFA: Nondeterministic execution where multiple states + coexist within a single context (Chapter VI) + +Chapter I Row Pattern Recognition Overview +============================================================================ + +Normative reference: ISO/IEC 19075-5 (SQL Technical Report, Part 5: Row +pattern recognition in SQL). Subclause numbers cited throughout this code +base refer to that document. Where Chapters 4 (FROM clause) and 6 (WINDOW +clause) describe parallel material, this implementation cites the Chapter 6 +subclause first because it targets Feature R020. + +Row Pattern Recognition (hereafter RPR) is a feature introduced in SQL:2016 +that matches regex-based patterns against ordered row sets. + +The SQL standard defines two forms: + + Feature R010: MATCH_RECOGNIZE (FROM clause) + - Dedicated table operator + - Provides dedicated functions such as MATCH_NUMBER(), CLASSIFIER() + - Supports ONE ROW PER MATCH / ALL ROWS PER MATCH + + Feature R020: RPR in a window (WINDOW clause) + - Integrated into the existing window function framework + - Supports ALL ROWS PER MATCH only + - No MATCH_NUMBER() + +This implementation targets Feature R020. + +The basic syntax is as follows: + + SELECT ... + OVER ( + PARTITION BY ... + ORDER BY ... + ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING + AFTER MATCH SKIP TO NEXT ROW | SKIP PAST LAST ROW + [INITIAL | SEEK] -- SEEK is defined in the standard but not implemented + PATTERN ( ) + DEFINE AS , ... + ) + +The PATTERN clause is a regular expression over row pattern variables. +The DEFINE clause specifies boolean conditions that determine whether each +variable evaluates to true for the current row. + +Example: + + PATTERN (A+ B) + DEFINE A AS price > PREV(price), + B AS price < PREV(price) + +This pattern matches "a span where prices rise consecutively then drop." + +PERMUTE is not supported (the parser raises an error). Its syntax is in the +grammar so that the standard spelling is diagnosed rather than read as a +pattern variable followed by a group; write the alternations out instead. + +Chapter II Overall Processing Pipeline +============================================================================ + +RPR processing is divided into three phases: + + +--------------------------------------------------------------+ + | 1. Parsing (Parser) | + | SQL text -> PATTERN parse tree + DEFINE expression tree | + | | + | 2. Compilation (Optimizer/Planner) | + | PATTERN parse tree -> optimization -> flat NFA elements | + | | + | 3. Execution (Executor) | + | Row-by-row matching via NFA simulation | + +--------------------------------------------------------------+ + +Each phase uses independent data structures, and the interfaces between +phases are well-defined: + + Parser -> Planner: WindowClause.rpPattern (RPRPatternNode tree) + WindowClause.defineClause (TargetEntry list) + + Planner -> Executor: WindowAgg.rpPattern (RPRPattern struct) + WindowAgg.defineClause (TargetEntry list) + +Chapter III Parsing Phase +============================================================================ + +III-1. Entry Point + + transformWindowDefinitions() (parse_clause.c) + +-- transformRPR() (parse_rpr.c) + +transformRPR() is invoked when RPCommonSyntax is present and performs the +following: + + (1) Frame option validation + - Only ROWS is allowed (RANGE, GROUPS are not) + - The start boundary must be CURRENT ROW + - EXCLUDE option is not allowed + - The end boundary must not be CURRENT ROW (UNBOUNDED FOLLOWING or a + positive offset FOLLOWING only) + + (2) Transcription to WindowClause + - Copies the rpPattern and rpSkipTo fields + + (3) DEFINE clause transformation (transformDefineClause) + +III-2. PATTERN parse tree + +The parser transforms the PATTERN clause into an RPRPatternNode tree. +Each node has one of the following four types: + + RPR_PATTERN_VAR Variable reference. Name stored in varName field. + RPR_PATTERN_SEQ Concatenation. Children node list in children. + RPR_PATTERN_ALT Alternation (or). Branch node list in children. + RPR_PATTERN_GROUP Group (parentheses). Body node list in children. + +All nodes have min/max fields to express quantifiers: + + A -> VAR(A, min=1, max=1) + A+ -> VAR(A, min=1, max=INF) + A* -> VAR(A, min=0, max=INF) + A? -> VAR(A, min=0, max=1) + A{3,5} -> VAR(A, min=3, max=5) + +If the reluctant field is true, the quantifier is reluctant (non-greedy). + +Example: PATTERN ((A+ B) | C*) + + ALT + +-- GROUP(1, 1) + | +-- SEQ + | +-- VAR(A, 1, INF) + | +-- VAR(B, 1, 1) + +-- VAR(C, 0, INF) + +Parentheses always produce a GROUP node; a GROUP(1, 1) like the one above is +unwrapped later, by Phase 1 (h). + +III-3. DEFINE Clause Transformation + +transformDefineClause() first validates the PATTERN variable count and +collects the names, then rejects any DEFINE variable that PATTERN does not +use. After that it processes each DEFINE variable as follows: + + (1) Checks for duplicate variable names + (2) Transforms the expression via transformExpr() and coerces it to + Boolean (coerce_to_boolean) right away, so that the steps below see + the final expression form + (3) Wraps in a TargetEntry with the variable name set in resname + (4) Extracts Var nodes via pull_var_clause() and ensures each is + present in the query targetlist, so the planner propagates the + referenced columns through the plan tree + +After all variables are processed: + (5) Validates navigation nesting and offsets (define_walker), marks + column origins and assigns collations + +Variables that are used in PATTERN but not defined in DEFINE are implicitly +evaluated as TRUE (matching all rows). + +Chapter IV Compilation Phase +============================================================================ + +IV-1. Entry Point + + create_windowagg_plan() (createplan.c) + +-- buildRPRPattern() NFA compilation (6 phases) + +IV-2. The 6 Phases of buildRPRPattern() + + Phase 1: parse tree optimization (optimizeRPRPattern) + Phase 2: Statistics collection (scanRPRPattern) + Phase 3: Memory allocation (makeRPRPattern) + Phase 4: NFA element fill (fillRPRPattern) + Phase 5: Finalization (finalizeRPRPattern) + Phase 6: Absorbability analysis (computeAbsorbability) + +IV-3. Phase 1: Parse Tree Optimization + +After copying the parser-generated parse tree, the following optimizations are +applied. + +A rewrite must keep not only the set of matchable lengths but also which +match is preferred. Several of them can move a repetition's exit decision +past a choice point inside the body, which leftmost-choice-first (7.2) makes +observable, so they are gated on the body consuming a fixed number of rows. +Alternatives of equal length count as fixed: they pick different variables, +never different rows. + + (a) SEQ flattening: Unwrap nested SEQ nodes + SEQ(A, SEQ(B, C)) -> SEQ(A, B, C) + + (b) Consecutive variable merging: Merge consecutive occurrences of the + same variable into a single quantifier + A A -> A{2} + A{2,3} A{1,2} -> A{3,5} + + (c) Consecutive group merging: Merge repeated identical groups whose + body consumes a fixed number of rows + (A B)+ (A B)+ -> (A B){2,INF} + (A | B B)+ (A | B B)+ stays as-is (how the iterations split between + the two groups is a choice the merged form does not have) + + (d) Consecutive ALT merging: Merge repeated identical ALT nodes + (A | B) (A | B) (A | B) -> (A | B){3} + + (e) Prefix/suffix merging: Merge an identical sequence before or after + a group. A prefix copy is mandatory and comes before the group, so + it always merges; a suffix copy merges only when the body consumes a + fixed number of rows. + A B (A B)+ -> (A B){2,INF} + + (f) ALT flattening and deduplication + (A | (B | C)) -> (A | B | C) + (A | B | A) -> (A | B) + + (g) Quantifier multiplication: Collapse nested quantifiers when the + achievable counts are contiguous and the preferred match does not + move + (A+)+ -> A+ + (A{2,3}){5} -> A{10,15} + (A{2,})* stays as-is (count 1 unreachable; A* would be wrong) + (A{2,3}){1,2} stays as-is (counts contiguous, but the nested form + settles the first iteration first and prefers three rows where + A{2,6} takes four) + + (h) Single-child unwrap + SEQ(A) -> A, (A){1,1} -> A + + (i) Reluctance normalization: a fixed count leaves reluctance nothing + to decide, so it is cleared wherever min == max + A{2}? -> A{2}, (A B){2}? -> (A B){2} + This runs before and after each node's own rewrites. (b) and (g) + decline a reluctant node outright, and (c), (d), (e) and (f) compare + nodes with rprPatternEqual(), which treats a reluctant node as unequal + to its greedy twin. Without the normalization (A{2}?){3} would stay + nested where (A{2}){3} collapses to A{6}, and (A{2}? | A{2}) would + keep both branches. The absorbability analysis of IV-5 reads the flag + as well, on a group's BEGIN and with no bound test, so a fixed-count + reluctant group becomes absorbable exactly where its greedy spelling + already was. + +IV-4. Phase 4: NFA Element Array Generation + +Transforms the optimized parse tree into a flat array of RPRPatternElement. +This is the core data structure used for NFA simulation at runtime. + +RPRPatternElement struct (16 bytes): + + Field Size Description + --------------------------------------------------------- + varId 1B Variable ID (0-0xEF) or control code (0xFB-0xFF) + depth 1B Group nesting depth + flags 1B Bit flags (see below) + reserved 1B Padding + min 4B Quantifier lower bound + max 4B Quantifier upper bound + next 2B Next element index (sequential flow) + jump 2B Branch link (ALT/SEP), group skip/loop-back (BEGIN/END) + +Pattern variables occupy varId 0 to RPR_VARID_MAX (0xEF) inclusive, +giving 240 distinct variables. Any varId with the high nibble set +(0xF0-0xFF) is reserved for control elements; 0xF0-0xFA are currently +spare. + +Control codes: + + RPR_VARID_BEGIN (0xFB) Group start marker + RPR_VARID_END (0xFC) Group end marker + RPR_VARID_ALT (0xFD) Alternation start marker + RPR_VARID_SEP (0xFE) Alternation branch separator + RPR_VARID_FIN (0xFF) Pattern completion marker + +Element flags (1 byte, bitmask): + + 0x01 RPR_ELEM_RELUCTANT (VAR, BEGIN, END) + Non-greedy quantifier. Prefers shorter match: try exit-loop + first, then repeat. Set on VAR for simple (A+?), + on BEGIN+END for group ((...)+?). + + 0x02 RPR_ELEM_EMPTY_LOOP (END) + Group body can produce empty match (all children nullable). + Creates a fast-forward exit clone alongside the normal + loop-back so cycle detection doesn't kill legitimate + matches. (IV-4b) + + 0x04 RPR_ELEM_EMPTY_PREFERRED (END) + Group body prefers the empty match over a consuming one. + Orders the fast-forward ahead of the loop-back below min, + where the group's own greed says nothing. (IV-4b) + + 0x08 RPR_ELEM_ABSORBABLE_BRANCH (VAR, BEGIN, END, ALT) + Element lies within an absorbable region. Used at runtime to + track whether the current NFA state is in an absorbable + context. See "IV-5. Absorbability Analysis" and + "VIII-2. Solution: Context Absorption" for more details about + absorption. + + 0x10 RPR_ELEM_ABSORBABLE (VAR, END) + Absorption comparison point. Where to compare consecutive + iterations for absorption. + - Simple unbounded VAR (A+): set on the VAR itself + - Unbounded GROUP ((A B)+): set on the END element only + + Accessor macros: + RPRElemIsReluctant(e) (e)->flags & 0x01 + RPRElemCanEmptyLoop(e) (e)->flags & 0x02 + RPRElemIsEmptyPreferred(e) (e)->flags & 0x04 + RPRElemIsAbsorbableBranch(e) (e)->flags & 0x08 + RPRElemIsAbsorbable(e) (e)->flags & 0x10 + +Roles of next and jump: + + - next: The next element to move to "after consuming" the current element. + For VAR, the next position after a successful match. + For BEGIN/END, the next position inside/outside the group. + For ALT, the first branch's content; for SEP, the next branch's + content (post-ALT on the last SEP). + + - jump: The element to "skip to." + In ALT, the first branch's terminating SEP. + In SEP, the branch link to the next branch's SEP (-1 on the last). + In BEGIN, a skip path to END+1 (for groups with min=0). + In END, a loop-back to the start of the group body. + +The examples below build up in order: a GROUP, then an ALT (which introduces +the SEP branch-separator markers), then the two combined. + +Example: PATTERN ((A B)+) -- GROUP + + idx varId depth min max next jump Description + -------------------------------------------------------------- + 0 BEGIN 0 1 INF 1 4 Group start + 1 A(0) 1 1 1 2 -1 A + 2 B(1) 1 1 1 3 -1 B + 3 END 0 1 INF 4 1 Group end + 4 FIN 0 1 1 -1 -1 Pattern completion + + - idx 0: BEGIN. next(=1) enters the group body. + jump(=4) skips to after END = FIN (used when min=0). + - idx 3: END. next(=4) exits the group. + jump(=1) loops back to the start of the group body. + +Example: PATTERN (A+ B | C) -- ALT + + Parse tree: ALT(SEQ(VAR(A,1,INF), VAR(B,1,1)), VAR(C,1,1)) + + Compilation result: + + idx varId depth min max next jump Description + ------------------------------------------------------------ + 0 ALT 0 1 1 1 3 Alternation start + 1 A(0) 1 1 INF 2 -1 Branch 1: A+ + 2 B(1) 1 1 1 6 -1 Branch 1: B -> FIN + 3 SEP 0 1 1 4 5 Branch 1 terminator + 4 C(2) 1 1 1 6 -1 Branch 2: C -> FIN + 5 SEP 0 1 1 6 -1 Branch 2 terminator (last) + 6 FIN 0 1 1 -1 -1 Pattern completion + + - idx 0: ALT marker. next(=1) enters branch 1's content, jump(=3) is + branch 1's terminating SEP + - idx 1: Variable A. next(=2) is B + - idx 2: Variable B. next(=6) is FIN (branch tail redirected past the ALT) + - idx 3: SEP. next(=4) enters branch 2's content, jump(=5) links to the + next branch's SEP + - idx 4: Variable C. next(=6) is FIN + - idx 5: SEP. jump(=-1) marks the last branch; next(=6) is post-ALT + - idx 6: FIN marker. Match completion signal + + Each branch is bounded by a trailing SEP, including the last, so a branch's + extent is fixed by its own SEP. The branch link runs from ALT through + the SEP chain, never through the branch content. That keeps it clear of a + branch that ends with a quantified group, whose BEGIN.jump is the group's + own skip-past-END path (redirected past the ALT so it never lands on a SEP). + +Example: PATTERN ((B C)* | A) -- GROUP + ALT combined + + idx varId depth min max next jump Description + -------------------------------------------------------------- + 0 ALT 0 1 1 1 5 Alternation start + 1 BEGIN 1 0 INF 2 8 Branch 1: (B C)* -- skip + 2 B(1) 2 1 1 3 -1 redirected from SEP(5) + 3 C(2) 2 1 1 4 -1 to post-ALT(8) + 4 END 1 0 INF 8 2 Branch 1 tail -> post-ALT + 5 SEP 0 1 1 6 7 Branch 1 terminator + 6 A(0) 1 1 1 8 -1 Branch 2: A -> post-ALT + 7 SEP 0 1 1 8 -1 Branch 2 terminator (last) + 8 FIN 0 1 1 -1 -1 Pattern completion + + A branch-terminal optional group's skip path: + + A group's BEGIN skip-past-END jump (taken when an optional group matches + zero times) is set to "the element after END". Here that element is the + branch's own SEP terminator (idx 5), so a naive skip would land on a SEP + marker. A SEP is a compile-time boundary, never a runtime state; landing + there would (via the default VAR switch arm) mis-treat the SEP's varId as an + always-true variable and consume a spurious row, or fall through SEP.next + into the next branch. + + fillRPRPatternAlt therefore redirects any branch-terminal BEGIN.jump that + points at the branch's SEP to the post-ALT element (so idx 1's jump is 8, + not 5), and the zero-match skip ends the branch -- an empty match, exactly + as it would outside an alternation. The same redirect covers the + last-branch case, where the element after END is the final SEP. + +IV-4a. Reluctant Flag (RPR_ELEM_RELUCTANT) + +The reluctant flag is set during Phase 4 (fillRPRPattern) from the parse +tree node's reluctant field. Phase 1 (i) has already cleared that field +wherever min == max, so A{2}? and (A B){2}? reach here as the plain +A{2} and (A B){2}. The flag reverses the priority of quantifier +expansion at runtime: + + Greedy (default): try loop-back first, then exit (prefer longer match) + Reluctant: try exit first, then loop-back (prefer shorter match) + +The flag is set on all elements that carry the quantifier: + + Simple VAR (A+?): RPR_ELEM_RELUCTANT on the VAR element + Group ((...)+?): RPR_ELEM_RELUCTANT on BEGIN and END elements + +At runtime (nfa_advance), the flag controls Depth-First Search +(DFS) exploration order: + + VAR with quantifier: (a VAR has no jump; looping stays on the element) + Greedy: primary path = stay (loop), clone = next (exit) + Reluctant: primary path = next (exit), clone = stay (loop) + + END element: + Greedy: primary path = jump (loop-back), clone = next (exit) + Reluctant: primary path = next (exit), clone = jump (loop-back) + + BEGIN with min=0: + Greedy: primary path = next (enter group), clone = jump (skip) + Reluctant: primary path = jump (skip), clone = next (enter group) + +The absorption optimization requires greedy quantifiers. Reluctant +quantifiers are excluded from absorbability analysis (see IV-5). + +IV-4b. Empty Match Flags (RPR_ELEM_EMPTY_LOOP, RPR_ELEM_EMPTY_PREFERRED) + +Both flags are set during Phase 4 (fillRPRPatternGroup) on the END element +and describe the group's body, not the group itself. fillRPRPattern* +returns them for every node and the END inherits the body's pair: a +concatenation carries a bit only when every child does (AND), an +alternation is nullable when any branch is (OR) but takes its preference +from the first branch alone, since leftmost-choice-first makes that the +preferred one. + +RPR_ELEM_EMPTY_LOOP -- the body is nullable, i.e. every path through it +can match zero rows: + + (A? B?)+ Both children nullable -> body nullable -> END gets flag + (A | B*)+ B* is nullable, making the ALT nullable -> END gets flag + +Both examples keep their BEGIN/END pair through Phase 1. A single nullable +child, as in (A?)*, is multiplied away by (g) into A*, and an unquantified +group is unwrapped by (h), so neither leaves an END to carry the flag. + +It marks the END for the cycle detection of IX-6, which only ever tests a +nullable END, and it lets nfa_advance_end offer a fast-forward exit beside +the loop-back below min. Without it, (A? B?){2,3} could not reach its lower +bound: iteration 1 consumes every available row, iteration 2 derives an +empty match, and nothing would carry the count to min(2). + +RPR_ELEM_EMPTY_PREFERRED -- the body's preferred derivation is the empty +one, which is what orders those two paths. A bare variable consumes a +row; only a reluctant quantifier that may take zero repetitions prefers to +skip it. The group's own greed cannot decide this: in ((A? B?){2}) min +equals max, so the group has no choice of iteration count left to be +greedy or reluctant about, while the body still prefers to consume rows. +The flag is what distinguishes ((A? B?){2}) from ((A?? B??){2}). +(See IX-4(c) for detailed runtime behavior.) + +IV-5. Absorbability Analysis (RPR_ELEM_ABSORBABLE) + +Context absorption is an optimization technique that reduces O(n^2) to O(n). +(Runtime behavior is described in Chapter VIII.) + +This phase determines whether the pattern has a structure suitable for the +absorption optimization and sets flags on the relevant elements: + + RPR_ELEM_ABSORBABLE Absorption comparison point + RPR_ELEM_ABSORBABLE_BRANCH Element within an absorbable region + +Eligibility conditions: + + (1) SKIP PAST LAST ROW (not NEXT ROW) + (2) Frame end is UNBOUNDED FOLLOWING + (3) No DEFINE variable depends on match_start (see VIII-3(c)) + +Structural conditions (isUnboundedStart + computeAbsorbabilityRecursive): + + Case 1: Simple VAR+ (e.g., A+) + -> ABSORBABLE | ABSORBABLE_BRANCH set on the VAR + Case 2: GROUP+ with fixed-length children (min == max, recursively) + e.g., (A B)+, (A B{2})+, ((A (B C){2}){2})+ + -> ABSORBABLE_BRANCH on all elements within the group, + ABSORBABLE | ABSORBABLE_BRANCH on END + + Why this is safe: when every child has min == max, the group + is semantically equivalent to unrolling its body into {1,1} + elements. E.g., (A B{2})+ behaves like (A B B)+. Each + iteration consumes a fixed number of rows, so an earlier + context's count always dominates a later one's (monotonicity). + + Case 3: GROUP+ whose body starts with VAR+ (e.g., (A+ B)+) + -> Recurses from BEGIN into the body, applying Case 1. + ABSORBABLE | ABSORBABLE_BRANCH set on A. + B and END get no flags -> absorption stops once past A. + +A reluctant group disqualifies its whole subtree. computeAbsorbability- +Recursive() returns at such a BEGIN, so (A+ B)+? gets no flags at all +even though A+ itself is greedy: the group prefers its shorter +alternative, which is the opposite of the ordering absorption relies on. + +Absorbability is determined per-element, not per-pattern. +Absorption comparison is performed only when a state resides at an +element with the RPR_ELEM_ABSORBABLE flag. Once a state leaves the +flagged region, absorption is permanently disabled for that state. + +Through this mechanism, the runtime guarantees monotonicity: +"a context that started earlier always subsumes a context that +started later." + +Chapter V NFA Runtime Data Structures +============================================================================ + +V-1. RPRNFAState -- NFA State + +A single NFA state represents "how far the pattern has progressed." + + Field Description + ----------------------------------------------------------- + elemIdx Index of the current pattern element + counts[] Repetition count per group depth + isAbsorbable Whether the state is in an absorbable region + next Next state in the linked list + +The size of the counts array is rpPattern->maxDepth (= maximum group +nesting depth + 1), allocated as a flexible array member at the end of +the struct. + +Example: In PATTERN ((A B)+ C), a state waiting for B in the 3rd iteration + + Element array: [0:BEGIN(d0) 1:A(d1) 2:B(d1) 3:END(d0) 4:C(d0) 5:FIN] + + elemIdx = 2 (B, depth 1) + counts[0] = 2 (depth 0: depth of END. Group completed 2 iterations) + counts[1] = 0 (depth 1: shared by A and B. A zeroed its own slot when it + exited, per the count-clear policy, so a state parked on B + always enters with zero) + + Counts are indexed by depth, not by elemIdx. + counts[0] is incremented when passing through END(depth 0), + and the group repetition count is preserved even when + the state is at B(depth 1). + +Definition of two states being "equal": + + Two states are equal if they have the same elemIdx and the same counts + up to the depth of that element. + nfa_states_equal() compares counts[0..elem->depth] using memcmp. + Only counts at or below the depth of the current element are meaningful. + +V-2. RPRNFAContext -- Matching Context + +A single context represents "a matching attempt started from a specific +start row." + + Field Description + --------------------------------------------------------------------- + states Linked list of active NFA states + matchStartRow Row number where matching started + matchEndRow Row number where matching completed + (-1 if incomplete) + lastProcessedRow Last row processed + matchedState State that reached FIN (for greedy fallback) + hasAbsorbableState Whether this context can absorb other contexts + allStatesAbsorbable Whether this context can be absorbed + next, prev Doubly-linked list + +Since the NFA is nondeterministic, multiple states can coexist +simultaneously within a single context. + +Example: In PATTERN ((A | B) C), if the first row matches both A and B, +two states coexist within the context: + + Element array: [0:ALT 1:A 2:SEP 3:B 4:SEP 5:C 6:FIN] + + State 1: elemIdx=5 (waiting for C, via branch A) + State 2: elemIdx=5 (waiting for C, via branch B) + +In this case, since the (elemIdx, counts) of the two states are equal, +nfa_add_state_unique() retains only State 1 (branch A), which was +added first. +Because DFS processes the first branch of ALT first, the state via A +is registered first, and the state via B is discarded as a duplicate. +This is the preferment guarantee. + +V-3. RPR Fields of WindowAggState + + nfaContext / nfaContextTail Doubly-linked list of active contexts + nfaContextFree Reuse pool for contexts + nfaStateFree Reuse pool for states + nfaVarMatched Per-row tri-state cache: varMatched[varId] (lazy) + nfaVisitedEnds Nullable ENDs reached in this DFS (cycle detection) + nfaVisitedMinWord Lowest bitmapword index touched since last reset + nfaVisitedMaxWord Highest bitmapword index touched since last reset + nfaStateSize Precomputed size of RPRNFAState + defineMatchStartDependent DEFINE vars needing per-context evaluation (match_start_dependent) + nfaLastProcessedRow Last row processed by NFA (-1 = none) + + EXPLAIN ANALYZE instrumentation counters are omitted here; see + execnodes.h for the full list. + +Memory management: + + States and contexts are managed through their own free lists. + Instead of palloc, they are obtained from the reuse pool, and + returned to the pool upon deallocation. + This reduces the overhead of frequent allocation/deallocation. + +Chapter VI NFA Execution: 3-Phase Model +============================================================================ + +VI-1. Entry Point and Overall Flow + +When the window function processes each row, row_is_in_reduced_frame() +is called. This function determines whether the current row belongs to +a matched frame, and if necessary, calls update_reduced_frame() to +drive the NFA. + +Flow of update_reduced_frame(): + + (1) Find or create a context for the target row + (2) Enter the row processing loop + (3) After the loop ends, record the match result + +Pseudocode of the row processing loop: + + targetCtx = ExecRPRGetHeadContext(pos) + if targetCtx == NULL: + targetCtx = ExecRPRStartContext(pos) + + for currentPos = startPos; targetCtx->states != NULL; currentPos++: + if not rpr_prepare_row(currentPos): -- row does not exist + ExecRPRFinalizeAllContexts() -- finalize all contexts + ExecRPRCleanupDeadContexts() -- clean up after finalization + break + + ExecRPRProcessRow(currentPos) -- 3-phase processing + ExecRPRStartContext(currentPos + 1) -- pre-create next start point + ExecRPRCleanupDeadContexts() -- remove dead contexts + +Key point: Processing a single row may require processing multiple rows +ahead. Due to the nature of window functions, determining the frame for +row N requires looking at rows beyond N. + +VI-2. Context Creation: ExecRPRStartContext() + +Creates a new context and performs the initial advance. + + (1) Allocate context via nfa_context_make() + (2) Set matchStartRow = pos + (3) Create initial state: elemIdx=0 (first pattern element), + counts=all zero + (4) Call nfa_advance() with currentPos = pos - 1 (no row consumed + yet) + +The initial advance expands epsilon transitions at the beginning of +the pattern. For example, the initial advance for PATTERN ((A | B) C): + + Start: elemIdx=0 (ALT) + -> Expand ALT branches + -> elemIdx=1 (A) -- VAR, so add state; stop here + -> elemIdx=3 (B) -- VAR, so add state; stop here + (element 2 is branch 1's SEP terminator) + + Result: Two states in the context {waiting for A, waiting for B} + +Reaching FIN during the initial advance is recorded like any other +match. Since currentPos is pos - 1, the record has matchEndRow < +matchStartRow: an empty match (RF_EMPTY_MATCH) if matchedState is set, +otherwise unmatched (RF_UNMATCHED). + +The quantifier flavors reach it differently. Reluctant min=0 (A*?, A??): +the skip path reaches FIN first and early termination prunes the enter +paths, so the empty match is final. Greedy (A*): the enter path adds its +VAR states before the skip path records FIN, so those states survive and +may match a longer span on a later row. + +VI-3. Row Preparation: rpr_prepare_row() + +Prepares the DEFINE evaluation context for the current row. DEFINE +predicates are NOT evaluated here; each variable is evaluated lazily the +first time the NFA consumes it (Phase 1 / nfa_eval_var_match), so a +variable that no active state tests at this row is never evaluated. + + fetch current row into temp_slot_1 -- return false if out of partition + set ecxt_outertuple = current row + invalidate nav_slot_pos + reset nfaVarMatched[] to RPR_VAR_UNEVALUATED + +nfaVarMatched is a tri-state array (RPRVarMatch): RPR_VAR_UNEVALUATED, +RPR_VAR_TRUE, or RPR_VAR_FALSE. nfa_eval_var_match() evaluates a +variable's DEFINE on first consumption and caches the result; a NULL +result folds to RPR_VAR_FALSE (non-True is not mapped). The caller +(advance_reduced_frame_nfa) holds winstate->currentpos at the scan +position for the whole row (restored after the loop) because the deferred +navigation opcodes read currentpos. + +To support row navigation operators (PREV, NEXT, FIRST, LAST), +a 1-slot model is used: only ecxt_outertuple is set to the current +row. Navigation is handled by EEOP_RPR_NAV_SET/RESTORE opcodes +emitted during DEFINE expression compilation: + + NAV_SET: save ecxt_outertuple, swap in target row via nav_slot + (evaluate): argument expression reads from swapped slot + NAV_RESTORE: restore original ecxt_outertuple + +Compound navigation (PREV(FIRST()), NEXT(FIRST()), PREV(LAST()), +NEXT(LAST())) is flattened by the parser into a single RPRNavExpr +with a compound kind (RPR_NAV_PREV_FIRST, etc.). The executor +computes the target position in two steps: first the inner reference +point (match_start + N or currentpos - N) with match-range validation, +then the outer adjustment (+/- M) with partition-range validation. +If either step is out of range, the result is NULL. + +nav_slot caches the last fetched position (nav_slot_pos) to avoid +redundant tuplestore lookups when multiple navigation calls target +the same row. + +The nfaVarMatched entries are filled lazily during Phase 1 (Match) as +variables are consumed. + +VI-4. Per-Context Invalidation (match_start_dependent variables) + +DEFINE variables that depend on match_start -- those containing FIRST or a +compound PREV_FIRST/NEXT_FIRST, or a LAST that carries an offset of its own, +whether plain or inside a compound PREV_LAST/NEXT_LAST -- are identified at +plan time via defineMatchStartDependent. For the head +context, advance_reduced_frame_nfa sets nav_match_start to its +matchStartRow before matching, so lazy evaluation uses the correct +FIRST/LAST base position. + +When processing a context whose matchStartRow differs, +nfa_reevaluate_dependent_vars() resets only the dependent variables to +RPR_VAR_UNEVALUATED so they are re-evaluated lazily against this context's +matchStartRow, installs nav_match_start to that value, and invalidates the +nav_slot cache. match_start-independent variables keep their cached value +across contexts (they do not read nav_match_start). + +nav_match_start is left installed and NOT restored: FIRST/LAST read it at +evaluation time, which happens later during nfa_match(); the next +context's invalidation or the next row's setup overwrites it. The +function also resets rprContext so one context's DEFINE scratch does not +accumulate across every context of a row. + +Summary of evaluation strategy by navigation content (a variable is +evaluated once per row and cached, except dependent ones which are +re-evaluated once per differing context): + + Navigation content evaluation + ------------------------------------------------------- + No navigation cached (once per row) + PREV/NEXT only cached (once per row) + LAST (no offset) cached (once per row) + LAST (with offset) per-context + FIRST (any) per-context + Compound (inner FIRST) per-context + Compound (inner LAST, no off.) cached (once per row) + Compound (inner LAST, w/off.) per-context + +VI-5. Tuplestore Mark and Trim (nodeWindowAgg.c) + +Navigation functions require access to past rows via the tuplestore. +To allow tuplestore_trim() to free rows that are no longer reachable, +the executor computes two offsets at init (see build_define_offsets): + + navMaxOffset (Nav Mark Lookback): + Maximum backward reach from currentpos. Contributed by PREV, + LAST (any offset, including the default 0), and compound + PREV_LAST/NEXT_LAST. + Mark position: currentpos - navMaxOffset. + + navFirstOffset (Nav Mark Lookahead): + Minimum forward reach from match_start. Contributed by FIRST + and compound PREV_FIRST/NEXT_FIRST. Can be negative when + compound PREV_FIRST looks before match_start. + Mark position: oldest_context->matchStartRow + navFirstOffset. + +The actual mark is set to: min(lookback_mark, lookahead_mark). +This ensures all rows reachable by any navigation function are retained. + +When offsets contain non-constant expressions (Param), the executor sets +navMaxOffsetKind/navFirstOffsetKind to RPR_NAV_OFFSET_NEEDS_EVAL. A constant +offset is resolved at init, as is a bind parameter the planner folded to a +Const for a custom plan; under a generic plan that parameter stays a Param and +resolves per scan, as a PARAM_EXEC offset does. Either way every navigation is +settled again per scan by resolve_nav_offsets(), which is where a null or +negative offset is rejected. On overflow, the kind is set to +RPR_NAV_OFFSET_RETAIN_ALL, disabling trim for that dimension. An offset that +resolves negative is rejected at execution, so that navigation can never run and +is left out of both reaches; it behaves exactly as if it were not in the DEFINE. +Each dimension is reported only when some navigation feeds it (hasMaxNav, +hasFirstNav), so an empty one prints nothing rather than a reach of zero. + +VI-6. ExecRPRProcessRow(): 3-Phase Processing + +NFA processing for a single row is divided into three phases: + + +--------------------------------------------+ + | Phase 1: MATCH (convergence) | + | Compare the current row against each VAR | + | state. Remove states that fail to match. | + | | + | Phase 2: ABSORB (absorption) | + | Merge duplicate contexts to prevent | + | state explosion. | + | | + | Phase 3: ADVANCE (expansion) | + | Expand epsilon transitions to prepare | + | for the next row. | + +--------------------------------------------+ + +This ordering is important: + + - Match executes first to "consume the current row." + - Absorb executes immediately after Match, when states have been updated. + - Advance executes last to prepare "states waiting for the next row." + +Chapter VII Phase 1: Match +============================================================================ + +nfa_match() iterates through each state in the context: + + (1) Check whether the state's elemIdx is a VAR element + (2) Compare against the current row using nfa_eval_var_match() + (3) Match success: increment repetition count, retain state + (4) Match failure: remove state + +Match determination (nfa_eval_var_match): + + If varId is within the range of defineClauseExprs: + Use the value of varMatched[varId] + + If varId exceeds the range (variable not defined in DEFINE): + Unconditionally true (matches all rows) + +Immediate advance to the comparison point: + + For a VAR inside an absorbable region -- one carrying + RPR_ELEM_ABSORBABLE_BRANCH without being the comparison point itself -- + that has no iteration left (count >= max) and whose next element is END, + the Match phase advances through the END chain immediately. + This is necessary for accurate state comparison in Phase 2 (Absorb). + + Example: In PATTERN ((A B)+), A stays where it is when it matches, since + its next element is B; when B matches, the state advances through END so + that the group count is complete for the absorption comparison with + other contexts. + +Chapter VIII Phase 2: Absorb (Context Absorption) +============================================================================ + +Absorption is the runtime optimization that collapses contexts which +have converged on identical future behavior. Two contexts are +treated as equivalent when one's bookkeeping (elemIdx and per-depth +iteration counts) is dominated by another's; the younger one is then +discarded. The optimization is safe because pattern matching is +monotonic -- an earlier context's reachable matches always contain a +later context's. This is what reduces the naive O(N^2) state count +to O(N). + +VIII-1. Problem + +In the current implementation, a new context is started for each row +processed. +Applying PATTERN (A+) to 10 rows produces 10 contexts, +each of which tracks state independently. + +If there are N rows, the total number of states becomes O(N^2): + + Context 1 (started at row 1): can match A up to N times + Context 2 (started at row 2): can match A up to N-1 times + ... + Context N (started at row N): can match A 1 time + +VIII-2. Solution: Context Absorption + +Key observation: a context started earlier contains +all matches of a later-started context (monotonicity principle). + +If Context 1 started at row 1 and matched A 5 times, +the state where Context 2 (started at row 2) matched A 4 times +is already contained within Context 1. + +Therefore Context 2 can be "absorbed" into Context 1. + +Worked example for PATTERN (A+) over 3 rows (each matches A): + + After row 1: + Ctx_1 (started row 1): state at A with counts[0] = 1 + + After row 2: + Ctx_1: state at A with counts[0] = 2 + Ctx_2 (started row 2): state at A with counts[0] = 1 + -> Same elemIdx; Ctx_1.count (2) dominates Ctx_2.count (1). + -> Ctx_2 absorbed. + + After row 3: + Ctx_1: state at A with counts[0] = 3 + Ctx_3 (started row 3): state at A with counts[0] = 1 + -> Ctx_1.count (3) dominates Ctx_3.count (1). + -> Ctx_3 absorbed. + +Total active contexts stays at O(1) instead of growing with N. + +The monotonicity argument covers only a context's future (in-progress) +matches, not a match it has already recorded (matchedState) -- e.g., one +on a non-absorbable branch, which an absorbing context cannot reproduce. +Absorption therefore excludes any context holding a recorded match (see +nfa_update_absorption_flags()). This costs no efficiency: SKIP PAST LAST +ROW still prunes such redundant contexts once the covering match is +recorded (nfa_add_matched_state()). + +VIII-3. Absorption Conditions + +Planner-time prerequisites (all must hold for absorption to be enabled): + + (a) SKIP PAST LAST ROW. SKIP TO NEXT ROW creates overlapping + contexts that cannot be safely absorbed. + (b) Unbounded frame (ROWS BETWEEN CURRENT ROW AND UNBOUNDED + FOLLOWING). Limited frames apply differently to each context, + breaking the monotonicity principle. + (c) No match_start_dependent navigation in DEFINE. + + Mechanism: each context has a different matchStartRow, so FIRST + resolves to a different row for each context at the same + currentpos. An earlier context's DEFINE result no longer + subsumes a later one's, making count-dominance comparison + invalid. Rather than comparing matchStartRow at runtime + (which would complicate the absorb path), any match_start + dependency disables absorption entirely. + + Navigation content match_start dep. absorption + ------------------------------------------------------------ + No navigation none safe + PREV/NEXT only none safe + LAST (no offset) none safe + LAST (with offset) boundary check unsafe + FIRST (any) direct unsafe + Compound (inner FIRST) direct unsafe + Compound (inner LAST, no off.) none safe + Compound (inner LAST, w/off.) boundary check unsafe + + The "match_start dep." column classifies how the navigation ties a + DEFINE result to the context's matchStartRow: + + none Independent of matchStartRow. The result depends + only on currentpos (or a fixed offset from it), so + every context evaluates it identically. + direct Computed from matchStartRow itself -- FIRST counts + forward from match start -- so the resolved row, + and thus the result, differs per context. + boundary check The resolved row is currentpos-relative (LAST with + a backward offset, or a compound whose inner LAST + carries an offset), but its in-range test is taken + against the match range [matchStartRow, currentpos]. + The range bound differs per context, so the result + can too. + + Only "none" is safe for absorption; "direct" and "boundary check" + both make an earlier context's result stop subsuming a later one's + (see (c) above). + +Runtime conditions (evaluated per context pair): + + (1) The pattern is marked as isAbsorbable (see IV-5) + (2) allStatesAbsorbable of the target context is true + (3) An earlier context "covers" all states of the target + +Cover condition (nfa_states_covered) -- "count-dominance": + + A state with the same elemIdx exists in the earlier context, + and the count at that depth is greater than or equal -- then it is + covered. The earlier context's per-depth iteration count thus + dominates the later one's; this is the count-dominance comparison + referenced in VIII-3(c). + +VIII-4. Dual-Flag Design + +Two boolean flags make the absorption decision efficient: + + hasAbsorbableState (monotonic: only true->false transition possible) + "Does this context have the ability to absorb other contexts?" + true if at least one absorbable state exists. + Transitions to false when states are removed leaving no absorbable + states. + Once false, it never becomes true again. + + allStatesAbsorbable (dynamic until a match is recorded) + "Can this context be absorbed?" + true if all states are in an absorbable region and no match is + recorded. + Becomes false when a non-absorbable state is added; reverts to true + when it is removed. Recording a match also sets it false and that + does not revert, since absorbing would free the match. + +VIII-5. Absorption Order + +nfa_absorb_contexts() traverses from tail (newest) to head (oldest). + + for ctx = tail to head: + if ctx.allStatesAbsorbable: + for older = ctx.prev to head: + if older.hasAbsorbableState: + if nfa_states_covered(older, ctx): + free(ctx) -- absorbed + break + +Since inspection starts from the newest context, the most recently started +(= having the shortest match) context is absorbed first. + +Chapter IX Phase 3: Advance (Epsilon Transition Expansion) +============================================================================ + +IX-1. Overview + +nfa_advance() expands epsilon transitions from each state after Match, +generating "new states waiting for the next row." + +An epsilon transition is a transition that moves without consuming a row: + + - ALT: branch to each alternative + - BEGIN: enter group (or skip if min=0) + - END: loop-back within group (or exit when condition is met) + - FIN: record match completion + - VAR loop/exit: repeat/exit according to the quantifier + +Expansion stops upon reaching a VAR element, and the state is added. +This is because VAR is the element that "will consume the next row." + +IX-2. Processing Order: DFS and Preferment + +advance processes states in lexicographic order, +performing Depth-First Search (DFS) on each state. + +This DFS order is what guarantees the SQL standard's "preferment": + + The branch that appears first in the PATTERN text takes precedence. + +Example: PATTERN (A | B) C + + The first branch A of the ALT takes precedence over the second branch B. + When both A and B can match, the match via A is selected. + +nfa_add_state_unique() prevents duplicate addition of the same state, +so the state added first (= from the preferred branch) is retained. + +IX-3. Routing Function: nfa_route_to_elem() + +Most inter-element transitions in the advance phase go through +nfa_route_to_elem(), but three callers reach nfa_advance_state() +directly: nfa_advance() (each state it pops), nfa_advance_alt() (its +per-branch clone), and nfa_route_to_elem()'s own skip path. The result +matches either way -- for a bypassed VAR, nfa_advance_var()'s count-zero +handling produces what park-and-skip would -- so "a VAR not yet matched" +is handled in two places. + +nfa_route_to_elem() branches on the type of the next element: + + If the next element is VAR: + (1) Add the state to the context (nfa_add_state_unique) + (2) If the VAR has min=0, also add a skip path (recurse via next). + A reluctant VAR (A??, A*?) reverses the order: the skip path goes + first, and the waiting state is dropped if it reaches FIN + -> Expansion stops here (VAR is the element that "will consume the next + row") + + If the next element is non-VAR (ALT, BEGIN, END, FIN): + -> Recursively call nfa_advance_state() to continue expansion + +With this structure, advance recursively follows epsilon transitions +until reaching a VAR, consistently stopping only at VAR elements. + +IX-4. Per-Element advance Behavior + +(a) ALT (nfa_advance_alt) + + Upon encountering an ALT element, branches are expanded in preference order + via the SEP branch-separator chain, stopping at the first branch that + records a match -- a later branch's FIN would replace the preferred one. + ALT.next is the first branch's content and ALT.jump is that branch's + terminating SEP; each SEP.jump links to the next branch's SEP (-1 on the + last) and each SEP.next is the next branch's content. + + ALT.next -> branch 1 content; ALT.jump -> SEP1 -> SEP2 -> ... (jump chain) + SEP_i.next -> branch (i+1) content + + nfa_advance_state() is recursively called at each branch's content. + +(b) BEGIN (nfa_advance_begin) + + Handles group entry. + jump points past the group: the element after END, or the post-ALT element + when the group ends an alternation branch (IV-4). + + BEGIN does not reset the count at its depth; it only asserts the slot + is already zero. Under the count-clear policy the previous occupant + of a depth slot clears it on the way out, so entry finds it clean. + See IX-4(c) and V-1. + + Greedy (default): + (1) Enter the group body (move via next) + (2) If min=0, also add a group skip path (move via jump) + + Reluctant: + Order reversed -- skip path first, group entry second. + If the skip path reaches FIN, the group entry path is not generated + (shortest match preferred). + +(c) END (nfa_advance_end) + + Handles group termination. This is the core of the repetition logic. + + Let count be the count at the current depth: + + count < min: + Loop-back (move via jump, repeat the group body) + + If the RPR_ELEM_EMPTY_LOOP flag is set: + In addition to loop-back, also add a fast-forward exit path. + This is because the body may produce an empty match, causing count + to never reach min. fast-forward resets counts[depth] to 0 + and exits via next (treating the remaining required iterations + as empty matches). + + The body decides which of the two comes first, not the group's own + greed: the fast-forward is explored first exactly when + RPR_ELEM_EMPTY_PREFERRED is set (IV-4b). If it then reaches FIN, + the loop-back is dropped, as in the min <= count < max arm below. + + min <= count < max: + Greedy: loop-back first, exit second + Reluctant: exit first, loop-back second + If the exit path reaches FIN, loop-back is omitted. + + count >= max: + Unconditional exit (move via next) + + On exit: reset counts[depth] = 0, and if the next element is an outer END, + increment the count at the outer depth. + +(d) VAR (nfa_advance_var) + + Handles repeat/exit for a VAR element with a quantifier. + + Let count be the count at the current depth: + + count < min: + Unconditional loop (stay at the same elemIdx, wait for the next row) + + min <= count < max: + Greedy: loop first, exit (next) second + Reluctant: exit first, loop second + If the exit path reaches FIN, loop is omitted. + + count >= max: + Unconditional exit (move via next) + + On exit: reset counts[depth] = 0. + +(e) FIN + + Match success. The current state is moved to matchedState for recording, + and matchEndRow is set to the current row. + + Upon reaching FIN, all remaining unprocessed states are removed + (early termination). By DFS order, the path that reached FIN first + has the highest preferment, so the rest are inferior paths. + This is the core mechanism that guarantees preferment. + + In SKIP PAST LAST ROW mode, upon reaching FIN, subsequent contexts + that started within the match range are immediately pruned. + +IX-5. State Deduplication: nfa_add_state_unique() + +When adding a new state to a context, it is compared against existing +states; +if an identical state already exists, it is not added. + +Comparison criteria: elemIdx + counts[0..elem->depth] (see V-1) + +This deduplication is the core mechanism that suppresses NFA state +explosion. +Because DFS order causes preferred-branch states to be added first, +identical states from lower-priority branches are automatically discarded. + +IX-6. Cycle Detection: nfaVisitedEnds + +When a group body can produce an empty match, +looping back from END may cause an infinite loop. + +Example: PATTERN ((A? B?)+) + + A? and B? both have min=0, so the body can pass through without + matching. If the group repeats: BEGIN -> A? skip -> B? skip -> END -> + BEGIN -> ... + +To prevent this: + + (1) At compile time: set the RPR_ELEM_EMPTY_LOOP flag on the END + of groups whose body is nullable. + The runtime effect of this flag is described in IX-4(c): + when count < min, a fast-forward exit path is added, + resolving the deadlock where count cannot increase due to empty + matches. + + (2) At runtime: initialize the nfaVisitedEnds bitmap immediately before + DFS expansion of each state within advance (once per state). + During DFS, nfa_advance_state marks an END carrying + RPR_ELEM_EMPTY_LOOP on entry, and nothing else. Reaching a marked + END means the body derived an empty match for this iteration -- a + DFS takes only epsilon transitions, so no row was consumed since + the last visit. The state is not discarded: it leaves the group + there, so that "leave the group" keeps its rank among the + alternatives (see IX-4(c)). + + Nothing else is marked, because nothing else can cycle. A revisit is + a cycle only when it carries no progress, and a state is really + (elemIdx, counts) -- the identity nfa_add_state_unique() compares. + A VAR consumes a row, and so does every derivation of a body that + cannot match empty; a loop-back into either is progress, not a cycle. + Marking them would discard legitimate re-entry and lose the match + outright: ((A | B B){1,3}){3} would then find no match at all. + + The guard is min-aware: a marked END exits only at count >= min (an + empty iteration at or above the lower bound stops the quantifier, + TR 19075-5 7.2.8). Below min the guard falls through to the normal + must-loop path instead, so the next iteration still runs and its + consuming branches park as usual -- a derivation that goes empty + first and consumes later stays reachable, and it is the preferred + one: ((A? | B){3} C) over rows {B},{A,C},{C} matches rows 1-2 by + going empty twice and taking B on the third iteration, not rows 1-3 + by taking B on the first. This terminates because every arrival at + an END increments its count, so the fall-through reaches min in + bounded steps and then exits through the guard. + + The count advances one empty iteration at a time, and each step is a + recursion, so the cost follows the lower bound -- which the grammar + caps only at RPR_QUANTITY_INF. The steps cannot be collapsed by + setting the count to the bound directly: the intermediate counts are + the enumeration order, and each one parks the body's consuming + branches at a rank of its own. ((A? | B){4} C) over rows + {B},{A},{A,C},{C} matches rows 1-3, which needs the loop-back at + count 3 specifically. + + That bound is on the count, not on the expansion. Within one + expansion only a nullable END is marked, so ALT and BEGIN may be + re-entered any number of times; where several alternations in a row + have nullable branches, the expansion enumerates paths instead of + states and costs 2^k for k such alternations. See the XXX at the + marking site in nfa_advance_state(). + + Marks are never cleared during a DFS. Clearing the body's marks on a + below-min loop-back would disarm the guard for any nested reluctant + unbounded quantifier inside the body, whose empty iterations then + recurse without bound: (A (B*?)+?){2,} on a single matching row. The + outer count does not advance while the inner one spins, so "the count + stops at min" is not on its own a termination argument. + +Chapter X Match Result Processing +============================================================================ + +X-1. Match Result + +RPR tracks the current match result as a single entry in WindowAggState +with two fields: rpr_match_start and rpr_match_length. When +rpr_match_start is >= 0 the entry describes the result for that position, +and rpr_match_length gives the kind: -1 for an unmatched row, 0 for an +empty match (pattern matched but consumed no rows), and >= 1 for a real +match of that many rows. When rpr_match_start is < 0, the position has +not been evaluated yet (RF_NOT_DETERMINED). + +A row's status against the current match result can be obtained by +calling get_reduced_frame_status(). + +X-2. AFTER MATCH SKIP + +Determines the starting point for the next match attempt after a successful +match: + + SKIP TO NEXT ROW: + New match attempt begins from the row after the match start row. + Overlapping matches are possible. + + SKIP PAST LAST ROW: + New match attempt begins from the row after the match end row. + Only non-overlapping matches are possible. + +X-3. INITIAL vs SEEK + + Standard definition (ISO/IEC 19075-5 6.12): + INITIAL: "is used to look for a match whose first row is R." + SEEK: "is used to permit a search for the first match anywhere + from R through the end of the full window frame." + In either case, if there is no match, the reduced window frame is empty. + The default is INITIAL. + + Current implementation: + SEEK is not supported (the parser raises an error). + Only INITIAL is supported, searching only for matches starting at each + row position pos. + +X-4. Bounded Frame Handling + + With RPR, the frame mode is always ROWS and the frame start must be + CURRENT ROW. The frame end must be UNBOUNDED FOLLOWING or a positive + offset (n >= 1) FOLLOWING; a CURRENT ROW end or a zero offset is + rejected, since it would reduce the frame to the single current row. + + When the frame is bounded (e.g., ROWS BETWEEN CURRENT ROW AND 5 + FOLLOWING), ExecRPRProcessRow receives hasLimitedFrame=true and + frameOffset indicating the upper bound. Before the match phase, + any context whose match has exceeded the frame boundary + (currentPos >= matchStartRow + frameOffset + 1) is finalized early + by forcing a mismatch. This prevents matches from extending beyond + the window frame. The sum is clamped to PG_INT64_MAX on overflow. + + Note that bounded frames also disable context absorption at the + planner level (see VIII-3(b)), since the frame boundary breaks the + monotonicity assumption required for correct absorption. + +Chapter XI Worked Example: Full Execution Trace +============================================================================ + +XI-1. Query + + SELECT company, tdate, price, + first_value(price) OVER w AS start_price, + last_value(price) OVER w AS end_price + FROM stock + WINDOW w AS ( + PARTITION BY company + ORDER BY tdate + ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING + AFTER MATCH SKIP PAST LAST ROW + PATTERN (A+ B) + DEFINE A AS price > PREV(price), + B AS price < PREV(price) + ); + +XI-2. Data + + Row# tdate price + -------------------------- + 0 2024-01-01 100 + 1 2024-01-02 110 + 2 2024-01-03 120 + 3 2024-01-04 115 + 4 2024-01-05 130 + +XI-3. Compilation Result + + PATTERN (A+ B) -> unchanged after optimization + + idx varId depth min max next jump + ----------------------------------------- + 0 A(0) 0 1 INF 1 -1 A+ + 1 B(1) 0 1 1 2 -1 B + 2 FIN 0 1 1 -1 -1 + + DEFINE: A -> "price > PREV(price)", B -> "price < PREV(price)" + isAbsorbable = true (A+ is a simple unbounded VAR) + +XI-4. Execution Trace + +The trace lists every variable's DEFINE value together for readability. In +the lazy model each variable is evaluated only when a state consumes it +(nfa_eval_var_match); a variable no state tests at a row stays +RPR_VAR_UNEVALUATED. + +--- Row 0 (price=100) --- + + update_reduced_frame(0) called. + + Context C0 created (matchStartRow=0). + Initial advance: elemIdx=0(A) -> VAR, so state is added. + C0.states = [{elemIdx=0, counts=[0]}] + + DEFINE values, row 0: + A: price(100) > PREV(price) -> no PREV -> false + B: price(100) < PREV(price) -> no PREV -> false + varMatched = [false, false] + + ExecRPRProcessRow(0): + Phase 1 (Match): A(0) state vs varMatched[0]=false -> state removed + C0.states = [] (empty) + + Phase 2 (Absorb): skipped (no states) + Phase 3 (Advance): skipped (no states) + + C0.states is empty, so the loop terminates. + matchEndRow < matchStartRow -> unmatched. + +--- Row 1 (price=110) --- + + update_reduced_frame(1) called. + + Context C1 created (matchStartRow=1). + Initial advance: C1.states = [{elemIdx=0, counts=[0]}] + + DEFINE values, row 1: + A: 110 > PREV(100) -> true + B: 110 < PREV(100) -> false + varMatched = [true, false] + + ExecRPRProcessRow(1): + Phase 1 (Match): A(0) match succeeds -> counts[0]++ -> counts=[1] + C1.states = [{elemIdx=0, counts=[1]}] + + Phase 3 (Advance): + State {elemIdx=0, counts=[1]}: A+ (min=1, count=1, max=INF) + count >= min, so: + Greedy -> loop first: keep {elemIdx=0, counts=[1]} + exit: reset counts[0]=0, next(=1) -> {elemIdx=1, + counts=[0]} + C1.states = [{elemIdx=0, counts=[1]}, {elemIdx=1, counts=[0]}] + +--- Row 2 (price=120) --- + + Context C2 created (matchStartRow=2). + Initial advance: C2.states = [{elemIdx=0, counts=[0]}] + + DEFINE values, row 2: + A: 120 > PREV(110) -> true + B: 120 < PREV(110) -> false + varMatched = [true, false] + + ExecRPRProcessRow(2): (each phase walks every context in turn) + Phase 1 (Match): + C1 {elemIdx=0, counts=[1]}: A matches -> counts=[2] + C1 {elemIdx=1, counts=[0]}: B does not match -> removed + C2 {elemIdx=0, counts=[0]}: A matches -> counts=[1] + C1.states = [{elemIdx=0, counts=[2]}] + C2.states = [{elemIdx=0, counts=[1]}] + + Phase 2 (Absorb): + Does C1 (started earlier) cover C2? + C1: {elemIdx=0, counts=[2]}, C2: {elemIdx=0, counts=[1]} + Same elemIdx, C1.counts >= C2.counts -> covered + C2 absorbed. -> removed. + + Phase 3 (Advance): + {elemIdx=0, counts=[2]}: Greedy -> loop + exit + Loop: {elemIdx=0, counts=[2]} + Exit: reset counts[0]=0, next(=1) -> {elemIdx=1, counts=[0]} + C1.states = [{elemIdx=0, counts=[2]}, {elemIdx=1, counts=[0]}] + + Context C3 created (matchStartRow=3). + +--- Row 3 (price=115) --- + + DEFINE values, row 3: + A: 115 > PREV(120) -> false + B: 115 < PREV(120) -> true + varMatched = [false, true] + + ExecRPRProcessRow(3): + Phase 1 (Match): + {elemIdx=0, counts=[2]}: A does not match -> removed + {elemIdx=1, counts=[0]}: B matches -> counts=[1] + C1.states = [{elemIdx=1, counts=[1]}] + + Phase 3 (Advance): + {elemIdx=1, counts=[1]}: B (min=1, max=1) + count(1) >= max(1) -> unconditional exit + Reset counts[0]=0, next = 2 (FIN) + FIN reached -> matchEndRow = 3, matchedState recorded. + Early termination: no remaining states, so completed immediately. + C1.states = [] (empty after reaching FIN) + + C1.states is empty and matchEndRow=3 >= matchStartRow=1 -> match succeeds. + + rpr_match_start = 1, rpr_match_length = 3 + +--- Row 4 (price=130) --- + + update_reduced_frame(4) called. + C3 was pruned when C1 recorded its match: under SKIP PAST LAST ROW every + context that started within the match's range is freed there. + New context C4 created (matchStartRow=4). + + DEFINE values, row 4: + A: 130 > PREV(115) -> true + B: 130 < PREV(115) -> false + + ... No subsequent rows, so ExecRPRFinalizeAllContexts() is called. + Match incomplete -> unmatched. + +XI-5. Final Result + + Row 0: unmatched -> reduced frame empty (window funcs NULL, count() 0) + Row 1: match head -> frame = rows 1 through 3 + Row 2: inside match -> skipped + Row 3: inside match -> skipped + Row 4: unmatched -> reduced frame empty (window funcs NULL, count() 0) + +Chapter XII Summary of Key Design Decisions +============================================================================ + +XII-1. Flat Array vs Tree-Based NFA + + The compiled pattern is stored as a flat array of fixed-size 16-byte + RPRPatternElement structs rather than as a tree. + + The array is contiguous and cache-friendly, elements reference each + other by 2-byte index instead of by pointer, and the whole structure + can be serialized with memcpy when passed to plan nodes. + +XII-2. Forward-only Execution vs Backtracking + + The NFA is simulated forward-only, tracking a set of live states, + rather than by backtracking. + + Backtracking would take exponential time in the worst case, whereas + forward-only NFA simulation is polynomial. Forward-only also fits the + window pipeline, which delivers sorted rows sequentially: it needs no + re-fetching of earlier rows, and each row's DEFINE conditions (SQL + expressions such as PREV or running aggregates, with high re-evaluation + cost) are evaluated once per row and cached; only match_start-dependent + variables are re-evaluated per context (VI-4). DFS order yields preferment + naturally, with greedy or reluctant behavior per quantifier obtained by + reversing that order. + +XII-3. Per-Context Management + + A separate match context is maintained for each start row. + + This supports overlapping matches under SKIP TO NEXT ROW, determines + each row's frame independently, and lets the absorption optimization + eliminate redundant contexts in O(n). + +XII-4. Memory Pool Management + + NFA states are managed through a custom free list, and both RPRNFAState + and RPRNFAContext are allocated in a partition-lifespan memory context + that is freed in release_partition. + + NFA states are created and destroyed in large numbers per row, so the + free list avoids palloc/pfree overhead. Their size varies (the + counts[] array), but maxDepth is fixed within a single query, so all + states have the same size. + +XII-5. Execution Optimization Summary + + The following optimizations make the NFA simulation practical. + + -- Compile-time -- + + (1) Parse Tree Optimization (IV-3) + + Simplifies the parse tree before converting the pattern to an NFA. + Reduces the number of NFA elements through consecutive variable + merging (A A -> A{2}), SEQ flattening, quantifier multiplication, + and other transformations. + + Significance: Reducing the element count directly shrinks the state + space, decreasing the cost of all subsequent runtime phases (match, + absorb, advance). + + -- Runtime: advance phase -- + + (2) Group Skip (IX-4(b)) + + At the BEGIN of a group with min=0, uses jump to skip the entire + group. Moves directly to the first element outside the group without + exploring the group body. Greedy enters then skips; Reluctant skips + then enters. + + Significance: For optional groups (min=0), immediately generates + a skip path without exploring the body, avoiding unnecessary DFS + expansion. + + (3) State Deduplication (IX-5) + + During advance, DFS may generate states with the same (elemIdx, + counts) combination through multiple paths. Additionally, for + group absorption, nfa_match performs inline advance from bounded + VARs (count >= max) within the absorbable region (ABSORBABLE_BRANCH) + through END chains to reach the comparison point (ABSORBABLE END). + This process can also produce duplicate states reaching the same END. + nfa_add_state_unique() blocks duplicate addition during advance. The + inline advance adds nothing -- it moves states in place -- so the + duplicates it leaves on an END are collapsed when the next advance + re-adds their successors. + + Significance: Prevents exponential growth of the state count in + ALT branches and quantifier expansion. Since DFS order causes the + preferred branch's state to be registered first, identical states + from lower-priority branches are automatically discarded, thereby + also guaranteeing preferment. + + (4) Cycle Detection and Fast-Forward (IX-6, IX-4(c)) + + When a nullable group body (e.g., A?) repeats empty matches, + the END -> BEGIN loop-back can continue indefinitely. + + Two mechanisms resolve this: + - A visited bitmap (nfaVisitedEnds) marks a nullable END whose body + has already derived an empty iteration. On a second arrival the + state leaves the group there once count >= min; below min it falls + through to the must-loop path, whose per-arrival count increment + reaches min in bounded steps (termination) + - At an END with the RPR_ELEM_EMPTY_LOOP flag set, when + count < min, the remaining required iterations are treated as + empty matches and a fast-forward exit path out of the group is + added (correctness) + + Significance: Cycle detection guarantees termination, and + fast-forward guarantees that the min condition is satisfied. + Without these, patterns containing nullable groups would fall + into infinite loops or fail to match. + + (5) Match Pruning (IX-4(e)) + + When a state reaches FIN during advance, all remaining unprocessed + states of that context are removed. Because of DFS order, the path + that reaches FIN first has the highest preferment, so the remaining + paths are inferior. + + Significance: Once the best match is determined, exploration of + inferior paths is immediately terminated. This mechanism achieves + both preferment guarantees and performance optimization. + + -- Runtime: inter-context -- + + (6) Early Termination (SKIP PAST LAST ROW) + + In SKIP PAST LAST ROW mode, when a match is found, subsequent + contexts whose start rows fall within the match range are pruned + immediately without further processing. + In SKIP TO NEXT ROW mode, overlapping contexts are preserved + because each row requires its own independent match. + + Significance: Prunes subsequent contexts whose start rows overlap + with a prior match range, avoiding unnecessary processing. + + (7) Context Absorption (Chapter VIII) + + If an independent context is created for each row, O(n^2) states + accumulate. By exploiting the monotonicity that an earlier-started + context subsumes the states of a later-started context, redundant + contexts are eliminated early. + + Absorbability is determined per-element; comparison is performed + only at elements with the RPR_ELEM_ABSORBABLE flag (see IV-5). + + Significance: Keeps the number of active contexts at a constant + level, achieving O(n^2) -> O(n) time complexity. Without this, + performance degrades sharply on long partitions. + +Appendix A. Data Structure Relationship Diagram +============================================================================ + + Parser Layer + -------- + RPCommonSyntax + |--- rpSkipTo: RPSkipTo + |--- rpDefs: List* (ResTarget) + +--- rpPattern: RPRPatternNode* (tree) + |--- nodeType: VAR | SEQ | ALT | GROUP + |--- min, max: quantifier + |--- reluctant: bool + |--- varName: variable name (VAR only) + +--- children: List* (SEQ/ALT/GROUP only) + + Planner Layer + ---------- + WindowAgg (plan node) + |--- rpSkipTo: RPSkipTo + |--- defineClause: List + +--- rpPattern: RPRPattern* + |--- numVars: int + |--- varNames: char** + |--- maxDepth: RPRDepth + |--- isAbsorbable: bool + |--- numElements: int + +--- elements: RPRPatternElement[] (flat array) + |--- varId (1B) + |--- depth (1B) + |--- flags (1B) + |--- reserved (1B) + |--- min, max (4B + 4B) + +--- next, jump (2B + 2B) + + Executor Layer + ---------- + WindowAggState + |--- rpSkipTo: RPSkipTo (AFTER MATCH SKIP mode) + |--- rpPattern: RPRPattern* (copied from plan) + |--- defineClauseExprs: List (DEFINE order, index == varId) + |--- nfaVarMatched: RPRVarMatch[] (per-row tri-state cache, lazy) + |--- defineMatchStartDependent: Bitmapset* (match_start_dependent + | DEFINE vars; see VI-4) + |--- nfaVisitedEnds: bitmapword* (cycle detection) + |--- nfaVisitedMinWord / nfaVisitedMaxWord: int16 + | (touched-word range for fast reset) + |--- nfaLastProcessedRow: int64 (-1 = none) + |--- nfaStateSize: Size (pre-calculated RPRNFAState allocation size) + |--- nfaContext <-> nfaContextTail (doubly-linked list) + | +--- RPRNFAContext + | |--- states: RPRNFAState* (linked list) + | | |--- elemIdx + | | |--- counts[] + | | +--- isAbsorbable + | |--- matchStartRow, matchEndRow + | |--- lastProcessedRow + | |--- matchedState (cloned on FIN arrival) + | |--- hasAbsorbableState + | +--- allStatesAbsorbable + |--- nfaContextFree (recycling pool) + +--- nfaStateFree (recycling pool) + +Appendix B. NFA Element Array Examples +============================================================================ + +B-1. PATTERN (A B C) + + idx varId depth min max next jump + ------------------------------------------ + 0 A 0 1 1 1 -1 + 1 B 0 1 1 2 -1 + 2 C 0 1 1 3 -1 + 3 FIN 0 1 1 -1 -1 + +B-2. PATTERN (A+ B*) + + idx varId depth min max next jump flags + ------------------------------------------------------------------------ + 0 A 0 1 INF 1 -1 ABSORBABLE | ABSORBABLE_BRANCH + 1 B 0 0 INF 2 -1 + 2 FIN 0 1 1 -1 -1 + + Only A+ is the absorption point (Case 1). Once past A, + absorption is permanently disabled for that state. + +B-3. PATTERN (A | B | C) + + idx varId depth min max next jump + ---------------------------------------- + 0 ALT 0 1 1 1 2 next -> branch 1, jump -> SEP1 + 1 A 1 1 1 7 -1 branch 1 -> post-ALT + 2 SEP 0 1 1 3 4 branch 1 term.; next -> B, jump -> SEP2 + 3 B 1 1 1 7 -1 branch 2 -> post-ALT + 4 SEP 0 1 1 5 6 branch 2 term.; next -> C, jump -> SEP3 + 5 C 1 1 1 7 -1 branch 3 -> post-ALT + 6 SEP 0 1 1 7 -1 branch 3 terminator (last) + 7 FIN 0 1 1 -1 -1 + + Each branch is terminated by a SEP. ALT.jump enters the SEP chain, each + SEP.jump links to the next branch's SEP (-1 on the last), and each SEP.next + enters the next branch's content; the branch tails are redirected to FIN. + +B-4. PATTERN ((A B)+ C) + + idx varId depth min max next jump flags + -------------------------------------------------------------------------- + 0 BEGIN 0 1 INF 1 4 ABSORBABLE_BRANCH + 1 A 1 1 1 2 -1 ABSORBABLE_BRANCH + 2 B 1 1 1 3 -1 ABSORBABLE_BRANCH + 3 END 0 1 INF 4 1 ABSORBABLE | ABSORBABLE_BRANCH + 4 C 0 1 1 5 -1 + 5 FIN 0 1 1 -1 -1 + + Case 2: GROUP+ with {1,1} body VARs. A, B are branches; + END is the absorption point. Compare with B-6 (Case 3). + +B-5. PATTERN ((A | B)+? C) + + idx varId depth min max next jump flags + ------------------------------------------------------------------- + 0 BEGIN 0 1 INF 1 7 RELUCTANT, group start + 1 ALT 1 1 1 2 3 next -> branch 1, jump -> SEP1 + 2 A 2 1 1 6 -1 branch 1 -> END + 3 SEP 1 1 1 4 5 branch 1 term.; jump -> SEP2 + 4 B 2 1 1 6 -1 branch 2 -> END + 5 SEP 1 1 1 6 -1 branch 2 terminator (last) + 6 END 0 1 INF 7 1 RELUCTANT, group end + 7 C 0 1 1 8 -1 + 8 FIN 0 1 1 -1 -1 + + The ALT lives inside a group, so its branch tails are redirected to the + post-ALT element (here the group's END at idx 6), not out of the group. + +B-6. PATTERN ((A+ B)+ C) -- Absorbability flag example + + idx varId depth min max next jump flags + --------------------------------------------------------------------------- + 0 BEGIN 0 1 INF 1 4 ABSORBABLE_BRANCH, group start + 1 A 1 1 INF 2 -1 ABSORBABLE | ABSORBABLE_BRANCH + 2 B 1 1 1 3 -1 + 3 END 0 1 INF 4 1 group end + 4 C 0 1 1 5 -1 + 5 FIN 0 1 1 -1 -1 + + Recurses from BEGIN into the body -> A matches Case 1 (simple VAR+). + A gets ABSORBABLE | ABSORBABLE_BRANCH, BEGIN gets ABSORBABLE_BRANCH. + B and END get no flags -> absorption stops once the state advances to B. + (See IV-5 Case 3) + +B-7. PATTERN ((A+ B | C*)+ D) -- Per-branch absorption in ALT + + idx varId depth min max next jump flags + --------------------------------------------------------------------------- + 0 BEGIN 0 1 INF 1 8 ABSORBABLE_BRANCH + 1 ALT 1 1 1 2 4 ABSORBABLE_BRANCH; jump -> SEP1 + 2 A 2 1 INF 3 -1 ABSORBABLE | ABSORBABLE_BRANCH + 3 B 2 1 1 7 -1 branch 1 -> END + 4 SEP 1 1 1 5 6 branch 1 term.; jump -> SEP2 + 5 C 2 0 INF 7 -1 ABSORBABLE | ABSORBABLE_BRANCH + 6 SEP 1 1 1 7 -1 branch 2 terminator (last) + 7 END 0 1 INF 8 1 EMPTY_LOOP + 8 D 0 1 1 9 -1 + 9 FIN 0 1 1 -1 -1 + + ALT branches are checked independently for absorbability. + Branch 1: A+ matches Case 1 -> A gets ABSORBABLE. B has no flag. + Branch 2: C* matches Case 1 -> C gets ABSORBABLE. + Both A and C get ABSORBABLE_BRANCH as part of their respective branch + paths. + END has EMPTY_LOOP: branch 2 (C*) is nullable, making the group body + nullable. + BEGIN and ALT get ABSORBABLE_BRANCH (on the path to absorbable elements). + The SEP branch-separator markers carry no flags: computeAbsorbabilityRecursive + walks the SEP chain but marks only branch content. + + +References: + +[1] ISO/IEC 19075-5 Information technology - Guidance for the use of + database language SQL - Part 5: Row pattern recognition + +[2] ISO/IEC 9075-2 Information technology - Database languages - SQL - + Part 2: Foundation (SQL/Foundation) + +============================================================================ + End of document +============================================================================ -- 2.43.0