Re: Logical replication and multimaster

On 3 December 2015 at 04:18, Konstantin Knizhnik <k(dot)knizhnik(at)postgrespro(dot)ru>
wrote:

> The problem is that transactions are delivered to replica through single
> channel: logical replication slot.
> And while such transaction is waiting acknowledgement from arbiter, it is
> blocking replication channel preventing other (parallel transactions) from
> been replicated and applied.
>

Streaming interleaved xacts from the slot as discussed in the prior mail
would help there.

You'd continue to apply concurrent work from other xacts, and just handle
commit messages as they arrive, sending the confirmations back through the
DTM API.

> I have implemented pool of background workers. May be it will be useful
> not only for me.
>

Excellent.

It should be possible to make that a separate extension. You can use C
functions from other extensions by exposing a single pg_proc function with
'internal' return type that populates a struct of function pointers for the
API. A single DirectFunctionCall lets you get the API struct. That's how
pglogical_output handles hooks. The main downside is that you can't do that
without a connection to a database with the extension installed so the
pg_proc entry is exposed.

So it could make more sense to just keep it as a separate .c / .h file
that's copied into trees that use it. Simpler and easier, but uglier.

> It consists of one produces-multiple consumers queue implemented using
> buffer in shared memory, spinlock and two semaphores.
>
[snip]

> You just place in this queue some bulk of bytes (work, size), it is placed
> in queue and then first available worker will dequeue it and execute.
>

Very nice.

To handle xact streaming I think you're likely to need a worker dispatch
key too, where the dispatch keys are "sticky" to a given worker. So you
assign xid 1231 to a worker at BEGIN. Send all work to the pool and
everything with xid 1231 goes to that worker. At commit you clear the
assignment of xis 1231.

Alternately a variant of the Execute method that lets you dispatch to a
specific worker would do the job.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services