Re: The plan for FDW-based sharding

On 02/24/2016 01:22 AM, Konstantin Knizhnik wrote:
> Sorry, but based on this plan it is possible to make a conclusion that
> there are only two possible cluster solutions for Postgres:
> XC/XL and FDW-based. From my point of view there are much more
> possible alternatives.

Definitely.

Currently we have five approaches to sharding inside postgres in the
field, in chronological order:

1. Greenplum's executor-based approach with motion nodes

2. Skype's function-based approach (PL/proxy)

3. XC/XL's approach, which I believe is also query executor-based

4. CitusDB's pg_shard which is based on query hooks

5. FDW-based (currently theoretical)

One of the things which causes bad reactions and arguments, Bruce, is
that a lot of your posts and presentations detailing plans for the FDW
approach carry the subtext that all four of the other approaches are
dead ends and not worth considering. Given that the other approaches,
whatever their limitations, have working code in the field and the FDW
approach does not, that's more than a little offensive.

If we want to move forwards on serious work on FDW-based sharding, the
folks working on it should stop treating it as a "fait accompli" that
this is the Chosen Way for the PostgreSQL project. Otherwise, you'll
spend all of your time arguing that point instead of working on features
that matter.

Bruce made a long comparison with built-in replication, but there's a
big difference here. We decided that WAL-based replication was the way
to go for built-in as a community decision here on -hackers and at
various conferences. Both the plan and the implementation for
replication transcended company backing, involving even active
competitors, and involved discussions with maintainers of the older
replication projects.

In contrast, this FDW plan *still* feels very much like a small group
made up of employees of only two companies came up with it in private
and decided that it should be the plan for the whole project. I know
that Bruce and others have good reasons for starting the FDW project,
but there hasn't been much of an attempt to obtain community consensus
around it. If Bruce and others want contributors to work on FDWs instead
of other sharding approaches, then they need to win over those people as
to why they should do that. It's how this community works.

Alternately, you can just work on the individual FDW features, which
*everyone* thinks are a good idea, and when most of them are done,
FDW-based scaleout will be such an obvious solution that nobody will
argue with it.

--
--
Josh Berkus
Red Hat OSAS
(any opinions are my own)