Re: Horizontal scalability/sharding

On Sun, Aug 30, 2015 at 5:31 PM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:

> On 30 August 2015 at 03:17, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
>
>> I have recently increased my public statements about the idea of adding
>> horizontal scaling/sharding to Postgres.
>
>
> Glad to see it. Many people have been pushing such things for years, so it
> is good to finally see some debate about this on Hackers.
>
>
>> I wanted to share with hackers
>> a timeline of how we got here, and where I think we are going in the
>> short term:
>>
>> 2012-2013: As part of writing my scaling talk
>> (http://momjian.us/main/presentations/overview.html#scaling), studying
>> Oracle RAC, and talking to users, it became clear that an XC-like
>> architecture (sharding) was the only architecture that was going to allow
>> for write scaling.
>>
>
> What other architectures were discussed? Where was that discussion?
>
>
>> Users and conference attendees I talked to were increasingly concerned
>> about the ability of Postgres to scale for high write volumes. They
>> didn't
>> necessarily need that scale now, but they needed to know they could get
>> it if they wanted it, and wouldn't need to switch to a new database in
>> the future. This is similar to wanting a car that can get you on a
>> highway
>> on-ramp fast --- even if you don't need it, you want to know it is there.
>>
>
> +1
>
>
>> 2014: I started to shop around the idea that we could use FDWs,
>> parallelism, and a transaction/snapshot manager to get XC features
>> as built-in to Postgres. (I don't remember where the original idea
>> came from.) It was clear that having separate forks of the source code
>> in XC and XL was never going to achieve critical mass --- there just
>> aren't enough people who need high right scale right now, and the fork
>> maintenance overhead is a huge burden.
>>
>
> I personally support the view that we should put scalability features into
> Postgres core, rather than run separate forks.
>
>
>> I realized that we would never get community acceptance to dump the XC
>> (or XL) code needed for sharding into community Postgres
>
>
> How or why did you realize that? There has never been any such discussion,
> AFAIK. Surely it can be possible to move required subsystems across?
>
>
>> , but with FDWs,
>> we could add the features as _part_ of improving FDWs, which would benefit
>> FDWs _and_ would be useful for sharding. (We already see some of those
>> FDW features in 9.5.)
>>
>
> That is a huge presumption. Not discussed or technically analyzed in any
> way with the community.
>
>
>> October, 2014: EDB and NTT started working together in the community
>> to start improving FDWs as a basis for an FDW-based sharding solution.
>> Many of the 9.5 FDW improvements that also benefit sharding were developed
>> by a combined EDB/NTT team. The features improved FDWs independent of
>> sharding, so they didn't need community buy-in on sharding to get them
>> accepted.
>>
>> June, 2015: I attended the PGCon sharding unconference session and
>> there was a huge discussion about where we should go with sharding.
>> I think the big take-away was that most people liked the FDW approach,
>> but had business/customer reasons for wanting to work on XC or XL because
>> those would be production-ready faster.
>>
>
> Cough, cough. You must surely be joking that "most people liked the FDW
> approach"? How did we measure the acceptance of this approach?
>
> What actually is the FDW approach? Since its not been written down
> anywhere, or even explained verbally, how can anyone actually agree to it?
>
>
>> July, 2015: Oleg Bartunov and his new company Postgres Professional (PP)
>> started to think about joining the FDW approach, rather than working on
>> XL, as they had stated at PGCon in June. A joint NTT/EDB/PP phone-in
>> meeting is scheduled for September 1.
>>
>
>

A little correction about Postgres Professional. We are concentrated on
idea to have one distributed transaction manager, originally DTM, now we
have better name XTM, which is neutral to actual cluster realization. For
example, we are testing it with XL, ported to 9.4, but we were planning to
extend tests to pg_shard, postgres_fdw. My idea was to have at least XTM
committed to 9.6, so all parties could work on their implementation much
easier.

> August, 2015: While speaking at SFPUG, Citus Data approached me about
>> joining the FDW sharding team. They have been invited to the September
>> 1 meeting, as have the XC and XL people.
>>
>
> 2ndQuadrant is working in this area, specifically bringing XL 9.5
> forwards. Please can invites be posted to myself, Pavan Deolasee and Petr
> Jelinek also? I'll pass on to others also.
>
> Koichi Suzuki is arranging a meeting in Hong Long for XC/XL discussions.
> Presumably EDB is invited also? If Koichi is a leading organizer of this,
> why are there two meetings?
>
> October, 2015: EDB is sponsoring a free 3-hour summit about FDW sharding
>> at the PG-EU conference in Vienna. Everyone is invited, but it is hoped
>> most of the September 1 folks can attend.
>>
>
>
>> February, 2016: Oleg is planning a similar meeting at their February
>> Moscow conference.
>>
>
>
>> Anyway, I wanted to explain the work that has been happening around
>> sharding.
>
>
> Thanks
>
>
>> As things move forward, I am increasingly convinced that write
>> scaling will be needed soon,
>
>
> +1
>
>
>> that the XC approach is the only reasonable way to do it,
>
>
>
>> and that FDWs are the cleanest way to get it into community
>> Postgres.
>>
>
> Those two things aren't at all obvious to me.
>
> Please don't presume my opposition. If the technical information were made
> public, I might understand and agree with "the FDW approach", perhaps
> others also. 2ndQuadrant is certainly happy to become involved in any team
> aiming to add features to Postgres core, as long as that makes sense. There
> may be areas we can all agree upon even if the full architecture remains in
> doubt.
>
> Before the community commits to a long term venture together we should see
> the plan. Like all IT projects, expensive failure is possible and the lack
> of a design is a huge flashing red warning light for me at present. If that
> requires a meeting of all Developers, why are the meetings for this
> specifically not happening at the agreed Developer meetings?
>
>
At PGCon we agreed to have such meeting in Vienna at least. But I think we
should be prepared and try to clean all our issues before. It looks like we
already out of time,but probably we could meet in Hong Kong ?

Honestly, I still don't know which approach is better, we already played
with XL (ported to 9.4) and identified some very strong issues with
inconsistency, which scared us, especially taking into account how easy we
found them. XC people have fixed them, but I'm not sure if they were
fundamental and if we could construct more sophisticated tests and find
more issues in XC/XL. We also a bit disappointed by Huawei position about
CSN patch, we hoped to use for our XTM. FDW approach has been actively
criticized by pg_shard people and that's also made me a bit suspicious. It
looks like we are doomed to continue several development forks, so we
decided to work on very important common project, XTM, which we hoped could
be accepted by all parties and eventually committed to 9.6. Now I see we
were right, unfortunately.

Again, could we organize meeting somewhere in September ? US is not good
for us, but other places should be ok. I want to have an agreement at
least on XTM. We still are testing various approaches, though. We could
present results of our experiments and are open to discussion. It's not
easy project, but it's something we could do for 9.6.

I'm very glad Bruce started this discussion in -hackers, since it's silly
to me to participate in both threads :) Let's meet in September !

> --
> Simon Riggs http://www.2ndQuadrant.com/
> <http://www.2ndquadrant.com/>
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
>