Re: The plan for FDW-based sharding

On 11 March 2016 at 16:09, Bruce Momjian <bruce(at)momjian(dot)us> wrote:

> Ideally everything would have a well-defined plan, it is
> sometimes hard to do.

BDR helped for logical decoding etc - having something concrete really
helped shape and guide each part of it as it was (or is/will be, in some
cases) migrated from BDR to core.

That said, it was necessary because for many of the things it needs there
weren't really good, isolated improvements to make with obvious utility for
other projects. Sure, commit timestamps are handy, replication origins will
be handy, etc. They can be used by other projects and will be. Some are
already. But unlike the FDW enhancements they're not things that will be
used simply by being present without even requiring any special user
action, so they had an understandably higher barrier to cross for
acceptance.

Once you get to the point where you're not making FDW improvements that
help a broad set of users and start doing things that'll really only aid
some hypothetical sharding system that also requires other infrastructure
changes, hooks, etc ... that's when I think it's going to be
proof-of-concept prototype time.

Similar to our approach on parallelism (which is
> also super-important and doesn't many active developers), sometimes you
> just need to create infrastructure and see how well it solves problems.
>

Yep. Again, like BDR and logical decoding. We've had quite a lot of
surprises as we find unexpected corner cases and challenges over time.
Andres's original work on logical decoding went through a number of
significant revisions as more was learned about the problem to solve.
Sometimes you can only do that by actually building it. Logical decoding as
it stands in core is only partway through that evolution as it is - I think
we now have a good understanding of why logical decoding of prepared xacts,
streaming of in-progress xacts etc will be needed down the track, but it
would've been hard to come up with that at the start when we didn't have
experience using what we've already got.

> The weird thing is that if you do implement an ill-defined feature,
> there really isn't much huge positive feedback --- people just use the
> feature, and the complaints stop.

... eventually.

Sometimes the bug reports start. Occasionally you get a "thanks, this looks
interesting/handy". But usually just bug reports or complaints that
whatever you built isn't good enough to meet some random person's
particular use case. Ah well.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services