RE: I'd like to discuss scaleout at PGCon

Hello, hackers

I'm very sorry to have left this thread for a long time. I've come out of hibernation, pushed by the need. Please let me resume the discussion on the scale-out design.

I'll read this thread and related ones again to refresh my memory, and I'd like to continue to assemble ideas and opinions in the following wiki page:

Scaleout Design - PostgreSQL wiki
https://wiki.postgresql.org/wiki/Scaleout_Design

I know there are lots of topics to decide in order to formulate the architectural design and functional specification. To get the most with minimal waste of efforts, I think we need to clarify what we want to achieve. Some of the most important topics are:

* What workloads do we target? Does Postgres give up on data warehousing capability that's comparable with existing products and cloud services?

* What architecture(s) do we want? Shared nothing, shared disk, a combination of them, or a totally new one.

* FDW or non-FDW approach? (as Simon and I mentioned in this thread, I don't think FDW is suitable for the scale-out, although we should try to reuse the code in postgres_fdw.)

From: Bruce Momjian <bruce(at)momjian(dot)us>
> On Sat, Jun 23, 2018 at 12:41:00PM +1000, Haribabu Kommi wrote:
> > Just I would like to bring out idea scale out by adding many instances that
> > can share the lock and buffer pool manager with all the instances with
> > the help of Remote direct memory access.
> >
> > By adding pluggable buffer pool and lock manager, how about adding
> > many instances and all share the buffers using RDMA to provide
> > better scaling with shared everything.
> >
> > Currently I didn't know have any idea whether is it possible or not and also
> > the problems in using RDMA.
> >
> > Just want to check whether is it worth idea to consider in supporting scale
> > out?
>
> Yes, Robert Haas did mention this. It might be something we consider
> much later.

I said that we wouldn't need shared disk scale-out at or around PGCon developer unconference 2018. But after that, I realized Oracle RAC users want shared disk scale-out for Postgres.

So, I wrote about the comparison of shared nothing and shared disk, and the rough design of shared disk scale-out as in the attached PDF file (I also attached it in the above wiki.) The attached file is what I used in PostgreSQL conference Japan 2019 to ask about how many users want shared disk scale-out. 25 out of 53 participants raised their hands to show their feelings of "want it" or "good to have." That was much more people than I had expected.

On the other hand, as I questioned in the last slide of the presentation, I'm not sure if we really need shared disk scale-out in this era that really powerful machines are available even on public clouds. Should we skip shared disk scale-out and just pursue single-node scale-up and shared nothing scale-out?

Amazon Aurora as an Alternative to Oracle RAC
https://aws.amazon.com/jp/blogs/database/amazon-aurora-as-an-alternative-to-oracle-rac/

"Stepping back and looking at the bigger picture, Amazon Aurora introduces a simplified solution that can function as an Oracle RAC alternative for many typical OLTP applications that need high performance writes, scalable reads, very high levels of high availability with lower operational overhead."

Migration Complete -- Amazon’s Consumer Business Just Turned off its Final Oracle Database | AWS News Blog
https://aws.amazon.com/jp/blogs/aws/migration-complete-amazons-consumer-business-just-turned-off-its-final-oracle-database/

Could you have a look at my presentation and give your opinion on whether we want shared disk scale-out for Postgres?

Regards
Takayuki Tsunakawa