Re: The plan for FDW-based sharding

On Tue, Mar 1, 2016 at 7:03 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:

> On Tue, Mar 1, 2016 at 10:37 AM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> > On Tue, Mar 1, 2016 at 10:19:45AM -0500, Robert Haas wrote:
> >> > Two reasons:
> >> > 1. There is no ideal implementation of DTM which will fit all
> possible needs
> >> > and be efficient for all clusters.
> >>
> >> Hmm, what is the reasoning behind that statement? I mean, it is
> >> certainly true that there are some places where we have decided that
> >> one-size-fits-all is not the right approach. Indexing, for example.
> >
> > Uh, is that even true of indexing? While the plug-in nature of indexing
> > allows for easier development and testing, does anyone create plug-in
> > indexing that isn't shipped by us? I thought WAL support was something
> > that prevented external indexing solutions from working.
>
> True. There is an API, though, and having pluggable WAL support seems
> desirable too. At the same time, I don't think we know of anyone
> maintaining a non-core index AM ... and there are probably good
> reasons for that. We end up revising the index AM API pretty
>

We'd love to develop new special index AM, that's why we all are for
pluggable WAL. I think there are will be other AM developers, once we open
the door for that.

> regularly every time somebody wants to do something new, so it's not
> really a stable API that extensions can just tap into. I suspect that
> a transaction manager API would end up similarly situated.
>

I don't expect many other TM developers, so there is no problem with
improving API. We started from practical needs and analyses of many
academical papers. We spent a year to play with several prototypes to prove
our proposed API (expect more in several months). Everybody could download
them a test. Wish we can do that with FDW-based sharding solution.

Of course, we can fork postgres as XC/XL people did and certainly
eventually will do, if community don't accept our proposal, since it's very
difficult to work on cross-releases projects. But then there are will be no
winners, so why do we all are aggressively don't understand each other ! I
was watching XC/XL for years and thought I don't want to go this way of
isolation from the community, so we decided to let TM pluggable to stay
with community and let everybody prove their concepts. if you have ideas
how to improve TM API, we are open, if you know it's broken by design,
let's help us to fix it. I have my understanding about FDW, but I
deliberately don't participate in some very hot discussion, just because I
feel myself not commited to work on. Your group is very enthusiastic on
FDW, it's ok until you improve FDW in general way, I'm very happy on
current work. I prefer you show prototype of sharding solution, which
convince us in functionality and perfromance. I agree with Thomas Vondra,
that we don't want to wait for years to see the result, we want to expect
results, based on prototype, which should be done between releases. If you
don't have enough resources for this, let's do together with community.
Nobody as I've seen are against FDW sharding, people complained about "the
only sharding solution" in postgres, without proving so.

>
> --
> Robert Haas
> EnterpriseDB: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>