Re: Horizontal scalability/sharding

On 2015-09-08 19:52, Robert Haas wrote:
> On Fri, Sep 4, 2015 at 6:52 PM, Ozgun Erdogan <ozgun(at)citusdata(dot)com> wrote:
>>> For Citus, a second part of the question is as FDW writers. We implemented
>> cstore_fdw, json_fdw, and mongo_fdw, and these wrappers don't benefit from
>> even the simple join pushdown that doesn't require Map/Reduce.
>>
>> The PostgreSQL wiki lists 85 foreign data wrappers, and only 18 of these
>> have support for joins:
>> https://wiki.postgresql.org/wiki/Foreign_data_wrappers
>
> What do you mean by "support for joins"? Do you mean that only 18 of
> the remote data sources can do joins? If so, why does that matter?
> I'd be quite happy if a join pushdown or "distributed shuffle" API had
> as many as 18 users - I'd be quite happy if it had one (postgres_fdw).
> The fact that not all FDWs can support every operation because of
> limitations on the remote side isn't a reason not to support those
> operations when the remote side is capable.
>

Agreed. While I personally don't think FDWs are long term answer for
sharding, I do believe that the infrastructure that is being developed
for FDWs (join/aggregate pushdown) is needed anyway and there are many
common issues that need solving in this area for FDWs, sharding and
parallel query.

--
Petr Jelinek http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services