Re: Horizontal scalability/sharding

On Sun, Aug 30, 2015 at 10:36:23PM +0300, Oleg Bartunov wrote:
> Honestly, I still don't know which approach is better, we already played with
> XL (ported to 9.4)  and identified some very strong issues with inconsistency,
> which scared us, especially taking into account how easy we found them. XC
> people have fixed them, but I'm not sure if they were fundamental and if we
> could construct more sophisticated tests and find more issues in XC/XL. We also
> a bit disappointed by Huawei position about CSN patch, we hoped to use for  our
> XTM.  FDW approach has been actively criticized by pg_shard people and that's
> also made me a bit suspicious. 

Yep, that has me concerned too. The pg_shard people will be on the
September 1 call and are working on a Google document to explain their
concerns about FDWs for sharding.

> It looks like  we are doomed to continue
> several development forks, so we decided to work on very important common
> project, XTM, which we hoped could be accepted by all parties and eventually
> committed to 9.6.  Now I see we were right, unfortunately.  

Yes, the ability to add independent parts that can eventually be used
for sharding is a strong indication that doing this incrementally is a
good approach.

> Again, could we organize meeting somewhere in September ?  US is not good for
> us, but other places should be ok. I want to have an agreement  at least on
> XTM. We still are testing various approaches, though. We could present results
> of our experiments and are open to discussion. It's not easy project, but it's
> something we could do for 9.6.

Good. XTM is a must-have for several use-cases, including sharding.

> I'm very glad Bruce started this discussion in -hackers, since it's silly to me
> to participate in both threads :)  Let's meet in September !

In summary, I think we need to start working on built-in sharding, and
FDWs are the only way I can see to do it with minimal code changes,
which I think might be a community requirement. It might not work, but
right now, it is the only possible approach I can see.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ Everyone has their own god. +