| From: | Etsuro Fujita <efujita(at)postgresql(dot)org> |
|---|---|
| To: | pgsql-committers(at)lists(dot)postgresql(dot)org |
| Subject: | pgsql: Support asynchronous execution of MergeAppend subplans. |
| Date: | 2026-11-17 06:38:04 |
| Message-ID: | E1xTCpP-000000004zt-3o2Y@gemulon.postgresql.org |
| Views: | Whole Thread | Raw Message | Download mbox | Resend email |
| Thread: | |
| Lists: | pgsql-committers |
Support asynchronous execution of MergeAppend subplans.
Commit 27e1f1456 added asynchronous execution for Append nodes, letting
multiple async-capable children -- in practice postgres_fdw ForeignScans
-- run concurrently instead of one at a time, but MergeAppend was not
covered.
This commit extends asynchronous execution to MergeAppend. On the first
call, requests for all valid async subplans are fired off up front, the
synchronous subplans are pulled while those requests are in flight, and
the async results are then collected before the merge heap is built.
After that, the merge cannot emit another row until the next tuple from
one specific subplan (the current heap top) is known, so at most one
request is outstanding during steady-state execution, and we always wait
indefinitely for it rather than polling as Append does.
Some other notable changes to the core and postgres_fdw code:
* create_merge_append_plan now marks children async-capable. Unlike a
plain Append, which disables async whenever its output must be
ordered, a MergeAppend always has pathkeys and still preserves
ordering by merging the per-child streams, so having pathkeys is not
disqualifying here; a child is eligible only if it is ordered well
enough on its own that no Sort node was inserted above it, as Sort is
not async-capable. The new GUC enable_async_merge_append (default on)
controls this.
* The event-loop body of ExecAppendAsyncEventWait is factored out into a
new ExecAsyncEventWait in execAsync.c and shared by both node types,
with the caller responsible for creating and freeing the WaitEventSet.
A new MergeAppendReady wait event reports waits in the new event loop.
* Since an FDW's ForeignAsyncConfigureWait callback can no longer assume
the requestor is an Append, add ExecAsyncGetRequestorEventSet to fetch
the requestor's WaitEventSet without knowing its node type, and use it
in postgres_fdw.
* postgresForeignAsyncConfigureWait also needs adjusting: the
optimization of skipping event configuration, rather than processing
an in-progress request made by a different requestor, is valid only
when our requestor is an Append; a MergeAppend cannot make progress
unless the given request produces a result, so in that case we process
the pending request and configure the event.
Full disclosure: I (Etsuro) reworked Alexander Pyhalov's original patch
with Claude Opus 4.8.
Author: Alexander Pyhalov <a(dot)pyhalov(at)postgrespro(dot)ru>
Author: Etsuro Fujita <etsuro(dot)fujita(at)gmail(dot)com>
Reviewed-by: Matheus Alcantara <matheusssilv97(at)gmail(dot)com>
Reviewed-by: Alexander Korotkov <aekorotkov(at)gmail(dot)com>
Reviewed-by: Alena Rybakina <a(dot)rybakina(at)postgrespro(dot)ru>
Reviewed-by: Álvaro Herrera <alvherre(at)kurilemu(dot)de>
Reviewed-by: solai v <solai(dot)cdac(at)gmail(dot)com>
Reviewed-by: Alexander Pyhalov <a(dot)pyhalov(at)postgrespro(dot)ru>
Reviewed-by: Etsuro Fujita <etsuro(dot)fujita(at)gmail(dot)com>
Discussion: https://postgr.es/m/59be194c5a409fb9fc9f2031581b8a44%40postgrespro.ru
Branch
------
master
Details
-------
https://git.postgresql.org/pg/commitdiff/11591f273e6a26fb55728b410538fa2627b19b28
Modified Files
--------------
contrib/postgres_fdw/expected/postgres_fdw.out | 442 +++++++++++++++++++++++-
contrib/postgres_fdw/postgres_fdw.c | 60 ++--
contrib/postgres_fdw/sql/postgres_fdw.sql | 119 ++++++-
doc/src/sgml/config.sgml | 14 +
doc/src/sgml/fdwhandler.sgml | 24 +-
doc/src/sgml/postgres-fdw.sgml | 10 +-
src/backend/executor/README | 15 +-
src/backend/executor/execAsync.c | 127 +++++++
src/backend/executor/nodeAppend.c | 94 +----
src/backend/executor/nodeMergeAppend.c | 301 +++++++++++++++-
src/backend/optimizer/path/costsize.c | 1 +
src/backend/optimizer/plan/createplan.c | 26 ++
src/backend/utils/activity/wait_event_names.txt | 1 +
src/backend/utils/misc/guc_parameters.dat | 7 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/executor/execAsync.h | 8 +
src/include/executor/nodeMergeAppend.h | 1 +
src/include/nodes/execnodes.h | 6 +
src/include/optimizer/cost.h | 1 +
src/test/regress/expected/sysviews.out | 3 +-
20 files changed, 1115 insertions(+), 146 deletions(-)
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Magnus Hagander | 2026-11-17 13:50:29 | Re: pgsql: ci: Enable PCH in mingw cross job |
| Previous Message | shihao zhong | 2026-11-17 01:43:57 | Re: pgsql: Don't trust NOT NULL on virtual generated columns of partitioned |