Re: Hooks to Modify Execution Flow and Query Planner

Hi,

On 2018/11/01 16:58, Vincent Mirian wrote:
> Hi all,
>
> I would like to create a library with UDFs written in C that implements
> different Query Planner tasks (e.g. scan, hash, join, etc...). I am looking
> for a document that provides an overview of execution flow within postgres
> and the query planner. I am also looking for a description of the software
> data structures and interfaces used.

Maybe, the following chapter in Postgres documentation will help as a start:

https://www.postgresql.org/docs/11/static/overview.html

For studying internal data structures and interfaces, you can also read
the comments contained in the source code and README files containing
descriptions of various data structures and interfaces, which is a often
recommended method.

Specifically, if you just want to inject alternative plan nodes for the
individual scan, hash, join operators needed to compute a query, but want
the Postgres query planner to take care of the whole planning itself, you
might consider looking into the Custom Scan Provider facility:

https://www.postgresql.org/docs/current/static/custom-scan.html

With it, you can write C code that gets invoked at certain points during
planning and execution, where you can add your special/custom node to a
plan and do execution related tasks on those nodes, respectively. With
this approach, Postgres planner and executor take care of most of the
details of planning and execution, whereas your code implements the
specialized logic you developed for, say, scanning a disk file, joining
two or more tables, building a hash table from the data read from a table,
etc.

You can alternatively formulate your code as a foreign data wrapper if all
you want do is model a non-Postgres data source as regular Postgres tables.

https://www.postgresql.org/docs/11/static/fdwhandler.html

If you don't intend to add new plan nodes or define a new type of foreign
table, but want to alter the planning or execution itself (or parts
thereof), you may want to look at various planner and executor hooks. For
example, if you want to replace the whole planner, which takes a parsed
query (the Query struct) and returns a plan (the PlannedStmt struct whose
internals you'll need to figure out if your alternative planning code can
produce a valid one), you can use the following hook:

/* Hook for plugins to get control in planner() */
typedef PlannedStmt *(*planner_hook_type) (Query *parse,
int cursorOptions,
ParamListInfo boundParams);

But that may be too complex a hook to implement on your own, so you can
look at more granular hooks which allow certain points within the
planning, such as:

/* Hook for plugins to get control in set_rel_pathlist() */
typedef void (*set_rel_pathlist_hook_type) (PlannerInfo *root,
RelOptInfo *rel,
Index rti,
RangeTblEntry *rte);

/* Hook for plugins to get control in add_paths_to_joinrel() */
typedef void (*set_join_pathlist_hook_type) (PlannerInfo *root,
RelOptInfo *joinrel,
RelOptInfo *outerrel,
RelOptInfo *innerrel,
JoinType jointype,
JoinPathExtraData *extra);

/* Hook for plugins to replace standard_join_search() */
typedef RelOptInfo *(*join_search_hook_type) (PlannerInfo *root,
int levels_needed,
List *initial_rels);

/* Hook for plugins to get control in get_relation_info() */
typedef void (*get_relation_info_hook_type) (PlannerInfo *root,
Oid relationObjectId,
bool inhparent,
RelOptInfo *rel);

On the executor side, you got:

/* Hook for plugins to get control in ExecutorStart() */
typedef void (*ExecutorStart_hook_type) (QueryDesc *queryDesc, int eflags);

/* Hook for plugins to get control in ExecutorRun() */
typedef void (*ExecutorRun_hook_type) (QueryDesc *queryDesc,
ScanDirection direction,
uint64 count,
bool execute_once);

/* Hook for plugins to get control in ExecutorFinish() */
typedef void (*ExecutorFinish_hook_type) (QueryDesc *queryDesc);

/* Hook for plugins to get control in ExecutorEnd() */
typedef void (*ExecutorEnd_hook_type) (QueryDesc *queryDesc);

/* Hook for plugins to get control in ExecCheckRTPerms() */
typedef bool (*ExecutorCheckPerms_hook_type) (List *, bool);

If you can be more specific about the what exactly you're trying to do,
someone can give even better advice.

Thanks,
Amit