Re: RFC: extensible planner state

On Wed, Aug 20, 2025 at 3:13 PM Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> Here's v2. 0001 is what you saw before with an attempt to fix the
> memory context handling. 0002 removes join_search_private. All I've
> tested is that the tests pass.

Here's v3 with a few more patches. I'm now fairly confident I have the
basic approach correct here, but let's see what others think.

0001 is the core "private state" patch for PlannerGlobal, PlannerInfo,
and RelOptInfo. It is unchanged since v2, and contains only the fix
for memory context handling since v1. However, I've now tested it, and
I think it's OK to commit, barring further review comments.

0002 removes join_search_private, as before. Whether it makes sense to
go ahead with this is debatable. Needs review, and needs an opinion on
whether this should be considered a PoC only (and discarded) or
something that should go forward to commit.

0003 adds two new planner hooks. In experimenting with 0001, I
discovered that it was a little hard to use. PlannerGlobal has to do
with what happens in a whole planning cycle, but the only hook we have
that's in approximately the right place is planner_hook, and it can't
see the PlannerGlobal object. So, I added these hooks. The first fires
after PlannerGlobal is fully initialized and before we start using it,
and the second fires just before we throw PlannerGlobal away. I
considered some other approaches, specifically: (1) making
subquery_planner a hook, (2) making grouping_planner a hook, and (3)
doing as the patch does but with the call before rather than after
assembling the PlannedStmt. Those proved inferior; the hook at the
very end of planner() just before we discard the PlannerGlobal object
appears quite valuable to me. Needs review.

0004 adds an extension_state member to PlannedStmt. Unlike the stuff
added by 0001, whatever goes into a PlannedStmt has to be a node tree.
The proposed usage convention is noted in the comment. There was
previous discussion of this kind of thing in
http://postgr.es/m/CA+TgmobrkCquFovDMZKRZ9cYQHnrS9sPE98aK0g2A=N1HFk3yQ@mail.gmail.com
and the messages leading up to it and I now believe this is exactly
the right way to enable what we were talking about over there: give a
plugin a chance to propagate whatever it likes from the PlannerGlobal
(including extension state) into the PlannedStmt, and then you can use
EXPLAIN hooks to print that stuff out -- or use it from anywhere that
has access to the PlannedStmt. Again, needs review.

0005 is a demo, not for commit, just to show how these pieces fit
together. It uses the hooks from 0001 to count the number of times
set_join_pathlist_hook is called and the number of those that are for
distinct joinrels. Then it uses planner_shutdown_hook to propagate
that into the PlannedStmt, and makes EXPLAIN (DEBUG) print those
values out. I think there are far more interesting bits of information
that could be preserved and propagated using this infrastructure,
though some of them probably also require other changes to make it all
work. But this is a simple example to show that the concept is valid
even without anything else.

For another example of how these patches could be used, see
http://postgr.es/m/CA+TgmoZ=6jJi9TGyZCm33vads46HFkyz6Aju_saLT6GFS-iFug@mail.gmail.com
and in particular 0001 and 0002. This patch set's planner_setup_hook
call would go write after those patches compute default_ssa_mask and
default_jsa_mask, allowing the hook to override those values. That's
not necessarily the very most interesting thing in the whole world,
because the real power of those patches is about manipulating ssa_mask
at the per-rel level and jsa_mask at the
per-call-to-add_paths_to_joinrel level; setting them for an entire
query isn't much better than we ca already do now by frobbing GUCs.
But it is a little better, because it allows automatically adjusting
the masks on a per-planner-invocation basis without regard to the
prevailing GUC values, so you could e.g. decide that whenever the
query ID has value X, we automatically set jsa_mask or ssa_mask to
value Y. Perhaps more interestingly, I think that planner_setup_hook
will prove to be the right place to set up a data structure at the
PlannerGlobal level that can be accessed by calls to
get_relation_info_hook and others to decide how to these masks should
be configured for each RelOptInfo.

I guess my point here is that I know this patch set (and the others
I've posted) seem a little thin in isolation, but the value starts to
compound when you think about them together. That's not to say that
I've got everything figured out here, only that I'd request that
nobody be too quick to dismiss any of these changes because they don't
do enough. The planner is extremely low on extension-author-friendly
infrastructure, and no single patch can or should try to solve that
problem completely.

Thanks,

--
Robert Haas
EDB: http://www.enterprisedb.com