Re: Future In-Core Replication

On Fri, Apr 27, 2012 at 1:26 AM, Josh Berkus <josh(at)agliodbs(dot)com> wrote:
> Simon,
>
>> I'm beginning to work on advanced additions to in-core replication for
>> PostgreSQL.
> ...
>> Those are the basic requirements that I am trying to address. There
>> are a great many important details, but the core of this is probably
>> what I would call "logical replication", that is shipping changes to
>> other nodes in a way that does not tie us to the same physical
>> representation that recovery/streaming replication does now. Of
>> course, non-physical replication can take many forms.
>
> So, I'm a bit confused.  You talk about this as "additions to in-core
> replication", but then you talk about implementing logical replication,
> which would NOT be an addition to Binary replication.  Can you explain
> what you mean?

The key point is that there is a specific objective of including
additional features in-core. That places some restrictions, but also
offers some opportunities. Tight integration allows performance
improvements, as well as ease of use etc..

I'm not sure what you mean by "would not be an addition to binary
replication". Yes, for reasons most elegantly explained by Robert here
[http://rhaas.blogspot.co.uk/2011/02/case-for-logical-replication.html],
physical/binary replication puts too many restrictions on us and we
cannot solve all of the problems that way. I was unaware of Robert's
post, but it sets the scene clearly.

So the future of in-core replication, IMHO, is some form of
non-physical replication. There are various options there, but
anything that goes in would reuse significant parts of the existing
replication setup that already works so well. Put that another way:
the infrastructure for the secure and efficient transport of
replication messages is already in place. Reuse is also what makes
something useful be achievable in a reasonable timescale.

What we need to consider is the form of those new non-physicalWAL
messages, how they are built on the sender and how they are handled at
the receiving end.

What I'm hoping to do is to build a basic prototype of logical
replication using WAL translation, so we can inspect it to see what
the downsides are. It's an extremely non-trivial problem and so I
expect there to be mountains to climb. There are other routes to
logical replication, with messages marshalled in a similar way to
Slony/Londiste/Bucardo/Mammoth(?). So there are options, with
measurements to be made and discussions to be had.

It will take time for people to believe this is possible and longer to
analyse and agree the options.

--
 Simon Riggs                   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training & Services