Re: Synchronous Standalone Master Redoux

On 07/10/2012 01:11 AM, Daniel Farina wrote:

> So if I get this straight, what you are saying is "be asynchronous
> replication unless someone is around, in which case be synchronous"
> is the mode you want.

Er, no. I think I see where you might have gotten that, but no.

> This is a pretty tricky definition: consider if you bring a standby
> on-line from archive replay and it shows up in streaming with pretty
> high lag, and stops all commit traffic while it reaches the bounded
> window of what "acceptable" lag is. That sounds pretty terrible, too.
> How does DBRD handle this? It seems like the catchup phase might be
> interesting prior art.

Well, DRBD actually has a very definitive sync mode, and no
"attenuation" is involved at all. Here's what a fully working cluster
looks like, according to /proc/drbd:

cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate

Here's what happens when I disconnect the secondary:

cs:WFConnection ro:Primary/Unknown ds:UpToDate/DUnknown

So there's a few things here:

1. Primary is waiting for the secondary to reconnect.
2. It knows its own data is still up to date.
3. It's waiting to assess the secondary when it re-appears
4. It's still capable of writing to the device.

This is more akin to degraded RAID-1. Writes are synchronous as long as
two devices exist, but if one vanishes, you can still use the disk at
your own risk. Checking the status of DRBD will show this readily. I
also want to point out it is *fully* synchronous when both nodes are
available. I.e., you can't even call a filesystem sync without the sync
succeeding on both nodes.

When you re-connect a secondary device, it catches up as fast as
possible by replaying waiting transactions, and then re-attaching to the
cluster. Until it's fully caught-up, it doesn't exist. DRBD acknowledges
the secondary is there and attempting to catch up, but does not leave
"degraded" mode until the secondary reaches "UpToDate" status.

This is a much more graceful failure scenario than is currently possible
with PostgreSQL. With DRBD, you'd still need a tool to notice the master
node is in an invalid state and perform a failover, but the secondary
going belly-up will not suddenly halt the master.

But I'm not even hoping for *that* level of functionality. I just want
to be able to tell PostgreSQL to notice when the secondary becomes
unavailable *on its own*, and then perform in "degraded non-sync mode"
because it's much faster than any monitor I can possibly attach to
perform the same function. I plan on using DRBD until either PG can do
that, or a better alternative presents itself.

Async is simply too slow for our OLTP system except for the disaster
recovery node, which isn't expected to carry on within seconds of the
primary's failure. I briefly considered sync mode when it appeared as a
feature, but I see it's still too early in its development cycle,
because there are no degraded operation modes. That's fine, I'm willing
to wait.

I just don't understand the push-back, I guess. RAID-1 is the poster
child for synchronous writes for fault tolerance. It will whine
constantly to anyone who will listen when operating only on one device,
but at least it still works. I'm pretty sure nobody would use RAID-1 if
its failure mode was: block writes until someone installs a replacement
disk.

--
Shaun Thomas
OptionsHouse | 141 W. Jackson Blvd. | Suite 500 | Chicago IL, 60604
312-444-8534
sthomas(at)optionshouse(dot)com

______________________________________________

See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email