Re: High Availability, Load Balancing, and Replication Feature Matrix

Markus Schiltknecht wrote:
> Hello Bruce,
>
> Bruce Momjian wrote:
> > Sorry, I forgot who was involved in that discussion.
>
> Well, at least that means I didn't annoy you to death last time ;-)

Certainly not. The more ideas the better. I need all the help I can
get.

> > Sorry, I meant that a master that is modifying data is slowed down by
> > other masters to an extent that doesn't happen in other cases (e.g. with
> > slaves). Is the current "No inter-server locking delay" OK?
>
> Yes, sort of. I know what you meant, but I find it hard to understand.
> And with regard to anything except lazy or eager replication, it does
> not make any sense. Its pretty moot saying anything about "inter-server
> locking delays" for "statement-based replication middleware": you don't
> know if it's lazy or eager. And all other solutions you are mentioning

I think the point is that with middleware each server is as least
working simultaneously while with multi-master they don't, at least in
most current implementations, no? Now you can end up being as slow as
the slowest server but that seems pretty hard to represent, no?

> are single-master or no replication at all. When there's only one
> master, it's pretty obvious that there can't be no inter-(master)-server
> locking delay. (Well, it's also very obvious that a single master never
> 'conflicts' with itself...)

Totally agree. What I need is a negative for multi-master so it is
clear why that option isn't used 100% of the time. The text above
clearly describes the reason, but how to do that in a bullet?

I was thinking I could take "No master server overhead" and somehow make
multi-master double-cost by using two bullets, but because it is a
negative I can't. :-( We could just remove "No inter-server locking
delay" and assume the "No master server overhead" represents the locking
overhead but that kind of loses the distinction that the multi-master
has much higher overhead. If you look at the chart it is kind of like
we have two items "no overhead" and "no significant overhead". Would
that be better?

> Given you want to cover existing solutions, one could say, that (AFAIK)
> all statement based replication solutions are eager. But in that case,

Agreed.

> the dot would be wrong, because the middleware would need to wait for at
> least an absolute majority to confirm the commit. Which as well leads to
> excessive locking, as you are saying for "synchronous multi-master
> replication". Because it's a property inherent to eager multi-master
> replication, as we correctly explain above the feature matrix.

See my comments above on simultaneous.

> >> multi-master replication" as well as "statement-based replication
> >> middleware" should not have a dot, because those as well slow down other
> >> masters. In the async case at different points in time, yes, but all
> >> master have to write the data, which slows them down.
> >
> > Yea, that is why I have the new text about locking.
>
> To me this makes it sound like "statement-based replication" could be
> faster than "synchronous multi-master replication". That's absolute
> nonsense, since those two don't compare. Or put it another way: most
> "statement-based replication" solutions often are "synchronous
> multi-master replication" as well.

Agreed "statement-based replication" in a way offers multi-master
capabilities, but as outlined above it has limitations as outlined in
the doc details. What I have done is changed the text to "No waiting
for multiple servers" and removed bullets from the appropriate
solutions. Is this better?

> >>> which is the reason we don't support it yet.
> >> Uhm.. PgCluster *is* a synchronous multi-master replication solution. It
> >> also is a middleware and it does statement based replication. Which dots
> >> of the matrix do you think apply for it?
> >
> > I don't consider PgCluster middleware because the servers have to
> > cooperate with the middleware.
>
> Okay, then take Sequoia: statement-based, middleware, synchronous (thus
> eager) multi-master replication solution.
>
> ( I've never liked the term "middleware" in that chapter. It's solely a
> question of implementation and does not have much to do with other
> concepts of replication. )

I had middleware in there because of the problem middleware has with
sequences and current_timestamp, i.e. you need to adjust the application
to deal with those sometimes.

> > I don't assume the disk failover has mirrored disks. It can just like a
> > single server can, but it isn't part of the backend process, and I
> > assume a RAID card that has RAM that can cache writes.
>
> In that case, you'd loose the "master failure will never lose data"
> property, no? Or do you trust the writeback cache and the connection to
> the NAS that much as to assume it never fails?

My assumption is that the _shared_ disk is not part of the master
itself. Of course if the shared disk fails you are out of luck, which
is mentioned above.

> >>> I don't think
> >>> the network is an issue considering many use NAS anyway.
> >> I think you are comparing an enterprise NAS to a low-cost, commodity
> >> hardware clustered filesystem. Take the same amount of money and the
> >> same number of mirrors and you'll get comparable performance.
> >
> > Agreed. In the one case you are relying on another server, and in the
> > NAS case you are relying on a black box server. I think the big
> > difference is that the other server is a separate entity, while the NAS
> > is a shared item.
>
> Correct, thus the former is a kind of single-master replication, while
> the later cannot be considered replication (lacking a replica). It's
> rather a variant of how to enhance reliability of your single-master
> database server system.

Right, which is why we call it "high availability" rather than
replication.

> >>> There is no dot there so I am saying "statement based replication
> >>> solution" requires conflict resolution. Agreed you could do it without
> >>> conflict resolution and it is kind of independent. How should we deal
> >>> with this?
> >> Maybe a third state: 'n/a'?
> >
> > Good idea, or "~". How would middleware avoid conflicts, i.e. how would
> > it know that two incoming queries were in conflict?
>
> A majority of servers rejecting or blocking the query? In case of a
> minority, which blocks, the majority would win and apply the
> transaction, while the minority would have to replay the transaction? I
> don't know, probably most solutions do something simpler, like aborting
> a transaction even if only one server fails. Much simpler, and
> sufficient for most cases.

Right, which I think we can call conflict resolution (abort on failure).

> (Why do you ask me, I'm advocating internal, tuple level replication
> with Postgres-R, not statement based one :-) )

Sure.

> > I did move it below and removed it from the chart because as you say how
> > to replicate to the slaves is an independent issue.
>
> Okay, I like that better, thanks.
>
> >> With regard to replication, there's another feature I think would be
> >> worth mentioning: dynamic addition or removal of nodes (masters or
> >> slaves). But that's solely implementation dependent, so it probably
> >> doesn't fit into the matrix.
> >
> > Yea, I had that but found you could add/remove slaves easily in most
> > cases.
>
> Hm.. you're right.
>
> >> Another interesting property I'm missing is the existence of single
> >> points of failures.
> >
> > Ah, yea, but then you get into power and fire issues.
>
> Which high-availability is all about, no?
>
> But well, again, all kinds of replication (which excludes the NAS) can
> theoretically be spread across the continent. So it might be pretty
> useless to add dots for that.

Yea.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +