Re: Append with naive multiplexing of FDWs

From: Bruce Momjian <bruce(at)momjian(dot)us>
To: Thomas Munro <thomas(dot)munro(at)gmail(dot)com>
Cc: pgsql-hackers <pgsql-hackers(at)postgresql(dot)org>, Stephen Frost <sfrost(at)snowman(dot)net>
Subject: Re: Append with naive multiplexing of FDWs
Date: 2019-09-27 16:20:02
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Wed, Sep 4, 2019 at 06:18:31PM +1200, Thomas Munro wrote:
> Hello,
> A few years back[1] I experimented with a simple readiness API that
> would allow Append to start emitting tuples from whichever Foreign
> Scan has data available, when working with FDW-based sharding. I used
> that primarily as a way to test Andres's new WaitEventSet stuff and my
> kqueue implementation of that, but I didn't pursue it seriously
> because I knew we wanted a more ambitious async executor rewrite and
> many people had ideas about that, with schedulers capable of jumping
> all over the tree etc.
> Anyway, Stephen Frost pinged me off-list to ask about that patch, and
> asked why we don't just do this naive thing until we have something
> better. It's a very localised feature that works only between Append
> and its immediate children. The patch makes it work for postgres_fdw,
> but it should work for any FDW that can get its hands on a socket.
> Here's a quick rebase of that old POC patch, along with a demo. Since
> 2016, Parallel Append landed, but I didn't have time to think about
> how to integrate with that so I did a quick "sledgehammer" rebase that
> disables itself if parallelism is in the picture.

Yes, sharding has been waiting on parallel FDW scans. Would this work
for parallel partition scans if the partitions were FDWs?

Bruce Momjian <bruce(at)momjian(dot)us>

+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +

In response to


Browse pgsql-hackers by date

  From Date Subject
Next Message Konstantin Knizhnik 2019-09-27 16:38:00 Re: Removing unneeded self joins
Previous Message Asif Rehman 2019-09-27 16:00:01 Re: WIP/PoC for parallel backup