Re: Timeline following for logical slots

On 6 April 2016 at 09:29, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:

> > Right now if you're doing any kind of logical deocoding from a master
> server
> > that fails over to a standby the client just dies. The slot vanishes.
> You're
> > stuffed. Gee, I hope you didn't need all that nice consistent ordering,
> > because you're going to have to start from scratch and somehow reconcile
> the
> > data in the new master with what you've already received ... and haven't.
>
> Right, but right now you probably *aren't* do doing any kind of
> logical decoding from a master server to a standby, because there's
> squat in the core distribution that could make use of that capability.
> So you never get as far as discovering this problem.
>

OK, I'm starting to understand where this is going.

pglogical and pglogical_output are irrelevant so long as they haven't yet
landed in core. There is no logical replication worth considering,
therefore anything that improves logical replication is not worth the
effort because there's really nothing to improve. Why bother with
enhancements like timeline following when nothing uses it.

Right?

That's pretty much what I'm hearing from a number of others too. Obviously
I don't wholly agree, but I do see where you're coming from. Especially
since pglogical is in a sort of twilight zone where it's neither a public,
open project with a proper website, public SCM, etc that encourages
external contribution, nor is it strictly and only a submission to core.

It's not like, say, PostGIS, in that it's always been aimed at core (or
stated to be) and is something that can be realistically considered for
core. It doesn't have much of a life as an independent extension, and at
least from the "outside" it doesn't look like there's much energy going
into giving it one, right? Yet it's also had limited time and effort put
into getting it in core in this release cycle.

While I don't like that, I have to say I understand it.

> First, it wasn't until 9.3 that physical standbys could follow
> timeline switches, but that doesn't mean that streaming replication
> was useless in 9.0 - 9.2

I don't think there's much of a parallel there. It's closer to being like
having _cascading_ replication without the ability to follow timeline
switches via WAL archive replay or streaming.

> Second, I'm not sure whether it was a good design decision to make
> logical slots a special kind of object that sit off to the side,
> neither configuration (like postgresql.conf) nor WAL-protected data
> (like pg_clog and the data files themselves), but it was certainly a
> very deliberate decision. I sort of expected them to be WAL-logged,
> but Andres argued (not unconvincingly) that we'd want to have slots on
> standbys, and making them WAL-logged would preclude that.

Yeah. I understand the reasons for that decision. Per an earlier reply I
think we can avoid making them WAL-logged so they can be used on standbys
and still achieve usable failover support on physical replicas.

> It's more like unlogged tables, where a deliberate
> design decision to lose data was made in order to meet some other
> goal.
>

Yeah ... but it's like unlogged tables without having the option of logged
tables.

Nice while it lasts but you're in bad trouble when it goes wrong.

(The parallel isn't perfect of course, since unlogged tables aren't crash
safe wheras slots are, they just aren't replicated).

> If you retype a column from text to integer,
> you probably aren't storing anything in it other than integers, in
> which case it is not necessarily the case that you are locked into
> applying that change at a particular point in the change stream.

... if you're replicating in text format, not send/recv format, yeah. You
get away with it in that particular case. But that *only* works where
you're using text format and the types are convertable via text
representation.

It's also just one example. Adding a NOT NULL column bites you, hard, for
example.

Asking users to do this themselves is incredibly fragile and requires much
more knowledge of PostgreSQL innards than I think is realistic. It's part
of why Slony-I is difficult too.

Sure, you can avoid having DDL replication if you're really, really careful
and know exactly what you're doing. I don't think it's a pre-req for
getting logical replication into core, but I do think it's a pretty strong
requirement for taking the use of logical replication for HA and failover
seriously for general use.

To be honest, I was shocked that pglogical and pglogical_output didn't
> go into this release. I assumed that you and other folks at
> 2ndQuadrant were going to make a big push to get that done. I did
> take a brief look at one of them - pglogical, I think - a week or two
> ago but there were unaddressed review comments that had been pending
> for months and there were a lot of fairly obvious things that needed
> to be done before it could be seriously considered as a core
> submission. Like, for example, rewriting the documentation heavily
> and making it look like the rest of our docs, and putting it in SGML
> format. The code seemed to need quite a bit of cleanup, too.

Yes, I agree, and I think those comments are largely fair. Both Petr and I
had more limited time to work on them than either of us had expected, so
timely follow-up was a challenge. We also got them ready for initial
submission later in the cycle than we'd hoped. That said, there was also
very little review response or interest for a long time after they were
published... and by the time there was I think both Petr and I were bogged
a bit in other project work.

In retrospect I think the separate submission of the output plugin was a
mistake too, at least when called pglogical_output; in part because of the
name it was completely ignored as meaningless without the downstream and
written off as useful for nothing except logical replication. So no
momentum was built by the time the downstream was published.

One thing that'd help a lot from my PoV would be if PGXS supported building
SGML docs. It's a major pain that maintaining a version as an extension (so
people can use it for older versions) requires a different documentation
format. But all in all that's relatively minor.

> Now,
> logical replication is a sufficiently important feature that if the
> only way it's going to get into core is if I work on it myself, or get
> other people at EnterpriseDB to do so, then I'll try to make that
> happen. But I was assuming that that was your/2ndQuadrant's patch,
> that you were going to get it in shape, and that me poking my nose
> into it wasn't going to be particularly welcome. Maybe I've misread
> the whole dynamic here.
>

I think getting more people involved is useful in a number of ways. Not
just time and resources, but also getting more agreement on the direction
things should be going in and making sure that more people have a direct
hand in the core and are happy with it. Working from a common repo where
interested people can push directly would be an interesting start.

I'll need to discuss this inside 2ndQ and follow up as there are things
involved that I can't make determinations for myself.

> > Do you want to get a logical replication system into core that doesn't
> work
> > properly with lots of the other features in PostgreSQL? That's
> historically
> > not how we've done things here, and sometimes massive amounts of work
> have
> > been required to make new feature X work with obscure/awkward existing
> > feature Y. Inheritance comes strongly to mind as an exciting challenge
> for
> > many new features to support properly. Arguments are usually raised that
> > reference things like the mess created by MySQL's storage engines and how
> > we're better off making everything work right once and for all.
>
> So, let me be clear, here. You have every right to decide which
> feature you want to work on, and if, at the current time, failover
> slots is that thing and if, at the current time, in-core logical
> replication solution is not that thing, then that is entirely up to
> you.

It's not entirely up to me personally, but I take your point.

My own priorities _have_ been influenced by how many people told me not to
bother with getting logical rep in core since it can just live as an
extension, though. I thought it reasonable to focus on things that can't be
done as an extension since pglogical looked like it had little hope of
landing in 9.6. Now I'm being told by others that it may as well not exist
until it's in core. That's ... a tad frustrating. But that's also part of
working in a community of individuals with different views and opinions.

Of course, I'm also biased to see pglogical as a going real-world concern
which is relevant whether in or out of tree, so to me it seems obviously
that improvements in core to make logical rep using it work better are
worthwhile. If the general viewpoint of the wider community is "not in
core, doesn't matter" then others will see it very differently.

FWIW, I'm also constantly keeping in mind that while I really, really want
pglogical in core, there'll certainly be a backport as an extension for
9.4/9.5 and now 9.6. Given the massive lead times involved in getting
anything into user hands I've really been hoping to get some of those
foundations into 9.6 so that the backport from 9.7+ works better on 9.6. I
suspect this either wasn't obvious or that others here don't see that as
interesting.

That being said, if we get a logical replication system into core that
> doesn't do DDL, doesn't do multi-master, doesn't know squat about
> sequences, and rolls over and dies if a timeline switch happens, I
> would consider that a huge step forward and I think a lot of other
> people would, too.

I can understand that.

I think it's pretty wrong for it to die on timeline switch. I also think
we'd really need *something* in place for sequences (even Londiste-style
sync-with-fudge-factor), but the rest is tolerable as far as I'm concerned.
I absolutely agree that multi-master isn't a requirement, and I think a
"queue some DDL for me" function is ugly but an acceptable workaround for
now. That's part of why pglogical's boundaries were set where they were
when separating it from BDR.

> We have a long history of building features
> incrementally. Parallel query in 9.6 doesn't parallelize every query
> that can be parallelized

Right. But it doesn't just arbitrarily break on what it doesn't understand.
That's the difference here.

> But
> plenty of people also make the opposite mistake, of thinking that they
> have to fix everything at once in order to have anything worthwhile,
> and that's just as much of a trap as the other thing.

Yeah, that's fair. Personally I think I pushed to cut pglogical to the bone
in terms of the minimum viable feature set that was targeted for it,
though. It goes very, very far from solving everything, and intentionally
ignores hard problems like DDL.

> Even with lots
> of limitations, built-in logical replication could still be good
> enough to be used long enough to manage a major version upgrade
>

... with some form of sequence handling added, yes. Not necessarily nice
sequence decoding, it can be worked around.

> Well, as noted above, I think the main thing we need to figure out is
> who is going to do the work. The main thing blocking other people
> from working on it is the belief that we are just waiting for you or
> someone else at 2ndQuadrant to land the necessary patches, but if you
> aren't really working on that in a focused way, then somebody else can
> step up.

It's a big job, and there are lots of differing viewpoints on it.

It does seem like having 2ndQ submit a patch, wait for feedback, be told
"oh, don't do it that way, come back with <something else>", then wait for
us to respond and address the issues isn't working. Certainly isn't going
to get us there in a timely manner.

I want to explore some options here, but I have to talk to internal team
members. Sorry for the handwavyness of that.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services