Re: asynchronous and vectorized execution

From: Kyotaro HORIGUCHI <horiguchi(dot)kyotaro(at)lab(dot)ntt(dot)co(dot)jp>
To: robertmhaas(at)gmail(dot)com
Cc: amitdkhan(dot)pg(at)gmail(dot)com, pgsql-hackers(at)postgresql(dot)org
Subject: Re: asynchronous and vectorized execution
Date: 2016-07-21 09:50:07
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers


At Tue, 12 Jul 2016 11:42:55 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI <horiguchi(dot)kyotaro(at)lab(dot)ntt(dot)co(dot)jp> wrote in <20160712(dot)114255(dot)156540680(dot)horiguchi(dot)kyotaro(at)lab(dot)ntt(dot)co(dot)jp>
> > 3% slower for local 1*seqscan (2-parallel)
> > 14% slower for append-4*seqscan (no-prallel)
> > 19% faster for append-4*foreignscan (all scans on one connection)
> > 78% faster for append-4*foreignscan (scans have dedicate connection)
> >
> > ExecProcNode might be able to be optimized a bit.
> > ExecAppend seems to need some fix.

After some refactoring, degradation for a simple seqscan is
reduced to 1.4% and that of "Append(SeqScan())" is reduced to
1.7%. The gains are the same to the previous measurement. Scale
has been changed from previous measurement in some test cases.

t0- (SeqScan()) (2 parallel)
pl- (Append(4 * SeqScan()))
pf0 (Append(4 * ForeignScan())) all ForeignScans are on the same connection.
pf1 (Append(4 * ForeignScan())) all ForeignScans have their own connections.

patched-O2 time(ms) stddev(ms) gain from unpatched (%)
t0 4121.27 1.1 -1.44
pl 1757.41 0.94 -1.73
pf0 6458.99 192.4 20.26
pf1 1747.4 24.81 78.39

t0 4062.6 1.95
pl 1727.45 9.41
pf0 8100.47 24.51
pf1 8086.52 33.53

> > Addition to the aboves, I will try reentrant ExecAsyncWaitForNode
> > or something.

After some consideration, I found that ExecAsyncWaitForNode
cannot be reentrant because it means that the control goes into
async-unaware nodes while having not-ready nodes, that is
inconsistent state. To inhibit such reentering, I allocated node
identifiers in depth-first order so that ascendant-descendant
relationship can be checked (nested-set model) in simple way and
call ExecAsyncConfigureWait only for the descendant nodes of the
parameter planstate.


Kyotaro Horiguchi
NTT Open Source Software Center

Attachment Content-Type Size
0001-Modify-PlanState-to-include-a-pointer-to-the-parent-.patch text/x-patch 21.4 KB
0002-Modify-PlanState-to-have-result-result_ready-fields..patch text/x-patch 67.4 KB
0003-Lightweight-framework-for-waiting-for-events.patch text/x-patch 15.2 KB
0004-Fix-async-execution-framework.patch text/x-patch 19.3 KB
0005-Add-new-fdwroutine-AsyncConfigureWait-and-ShutdownFo.patch text/x-patch 3.6 KB
0006-Make-postgres_fdw-async-capable.patch text/x-patch 40.6 KB
0007-Make-Append-node-async-aware.patch text/x-patch 4.5 KB

In response to


Browse pgsql-hackers by date

  From Date Subject
Next Message Kyotaro HORIGUCHI 2016-07-21 09:52:49 Re: [BUG] pg_basebackup from disconnected standby fails
Previous Message Kyotaro HORIGUCHI 2016-07-21 08:19:50 Re: [BUG] pg_basebackup from disconnected standby fails