Re: The plan for FDW-based sharding

On Tue, Mar 1, 2016 at 7:03 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:

> On Tue, Mar 1, 2016 at 10:37 AM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> > On Tue, Mar 1, 2016 at 10:19:45AM -0500, Robert Haas wrote:
> >> > Two reasons:
> >> > 1. There is no ideal implementation of DTM which will fit all
> possible needs
> >> > and be efficient for all clusters.
> >>
> >> Hmm, what is the reasoning behind that statement? I mean, it is
> >> certainly true that there are some places where we have decided that
> >> one-size-fits-all is not the right approach. Indexing, for example.
> >
> > Uh, is that even true of indexing? While the plug-in nature of indexing
> > allows for easier development and testing, does anyone create plug-in
> > indexing that isn't shipped by us? I thought WAL support was something
> > that prevented external indexing solutions from working.
>
> True. There is an API, though, and having pluggable WAL support seems
> desirable too. At the same time, I don't think we know of anyone
> maintaining a non-core index AM ... and there are probably good
> reasons for that.

It's because we didn't offer legal mechanism for pluggable AMs.

> We end up revising the index AM API pretty
> regularly every time somebody wants to do something new, so it's not
> really a stable API that extensions can just tap into.

I can't buy this argument. One may say this about any single API. Thinking
so will lead you to rejecting any extendability. And that would be in
direct contradiction to original postgres concept.
During last 5 years we add 2 new AMs: SP-GiST and BRIN. And BRIN is very
different from any other AM we have before.
And I wouldn't say that AM API have dramatical changes during that time.
There were some changes. But it would be normal work for extension
maintainers to adopt these changes like they do for other API changes.

There is simple example where we lack of extensible AMs: fast full-text
search. We can't provide it with current GIN, because we lack of positional
information in it. And we can't push these advances into core because
current implementation has not perfect design. Ideal design would be push
all required functionality into btree, then make GIN wrapper over btree
then add required functionality. But this is roadmap for 5-10 years. These
5-10 years uses will suffer from having 3-rd party solutions for fast FTS
instead of in core one. But our design questions is actually not something
that users care about. It's not reliability questions. And having pluggable
AMs would be really chance in this situation. Users could use extension
right now. And then when after many years we finally implement the right
design, they could migrate to in-core solution. But 5-10 years of fast FTS
does matter.

> I suspect that
> a transaction manager API would end up similarly situated.
>

I disagree with you about AM API. But I agree that TM API should be in
similar situation to AM API.

------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company