Async execution of postgres_fdw.

From: Kyotaro HORIGUCHI <horiguchi(dot)kyotaro(at)lab(dot)ntt(dot)co(dot)jp>
To: pgsql-hackers(at)postgresql(dot)org
Subject: Async execution of postgres_fdw.
Date: 2014-12-15 09:09:44
Message-ID: 20141215.180944.242184148.horiguchi.kyotaro@lab.ntt.co.jp
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hello, this is the 2nd session of 'intoroducing parallelism using
postgres_fdw'.

The two patch attached are as following,

- 0001-Async-exec-of-postgres_fdw.patch
Main patch, which includes all functions.

- 0002-rename-PGConn-variable.patch
Renaming the variable conn for readability. No functional
effect.

* Outline of this patch

From some consideration after the previous discussion and
comments from others, I judged the original (WIP) patch was
overdone as the first step. So I reduced the patch to minimal
function. The new patch does the following,

- Wrapping PGconn by PgFdwConn in order to handle multiple scans
on one connection.

- The core async logic was added in fetch_more_data().

- Invoking remote commands asynchronously in ExecInitForeignScan.

- Canceling async invocation if any other foreign scans,
modifies, deletes use the same connection.

Cancellation is done by immediately fetching the return of
already-invoked acync command.

* Where this patch will be effective.

With upcoming inheritance-partition feature, this patch enables
stating and running foreign scans asynchronously. It will be more
effective for longer TAT or remote startup times, and larger
number of foreign servers. No negative performance effect on
other situations.

* Concerns about this patch.

- This breaks the assumption that scan starts at ExecForeignScan,
not ExecInitForeignScan, which might cause some problem.

- error reporting code in do_sql_command is quite ugly..

regards,

--
Kyotaro Horiguchi
NTT Open Source Software Center

Attachment Content-Type Size
0001-Async-exec-of-postgres_fdw.patch text/x-patch 26.3 KB
0002-rename-PGConn-variable.patch text/x-patch 13.9 KB

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Samrat Revagade 2014-12-15 09:16:02 Re: pg_rewind in contrib
Previous Message Petr Jelinek 2014-12-15 08:41:36 Re: tracking commit timestamps