Re: Logical decoding restart problems

On 19/08/16 09:34, konstantin knizhnik wrote:
>
> We are using logical decoding in multimaster and we are faced with the
> problem that inconsistent transactions are sent to replica.
> Briefly, multimaster is using logical decoding in this way:
> 1. Each multimaster node is connected with each other using logical
> decoding channel and so each pair of nodes
> has its own replication slot.
> 2. In normal scenario each replication channel is used to replicate only
> those transactions which were originated at the source node.
> We are using origin mechanism to skip "foreign" transactions.
> When offline cluster node is returned back to the multimaster we need
> to recover this node to the current cluster state.
> Recovery is performed from one of the cluster's node. So we are using
> only one replication channel to receive all (self and foreign) transactions.
> Only in this case we can guarantee consistent order of applying
> transactions at recovered node.
> After the end of recovery we need to recreate replication slots with all
> other cluster nodes (because we have already replied transactions from
> this nodes).
> To restart logical decoding we first drop existed slot, then create new
> one and then start logical replication from the WAL position 0/0
> (invalid LSN).
> In this case recovery should be started from the last consistent point.
>

I don't think this will work correctly, there will be gap between when
the new slot starts to decode and the drop of the old one as the new
slot first needs to make snapshot.

Do I understand correctly that you are not using replication origins?

> The problem is that for some reasons consistent point is not so
> consistent and we get partly decoded transactions.
> I.e. transaction body consists of two UPDATE but reorder_buffer extracts
> only the one (last) update and sent this truncated transaction to
> destination causing consistency violation at replica. I started
> investigation of logical decoding code and found several things which I
> do not understand.

Never seen this happen. Do you have more details about what exactly is
happening?

>
> Assume that we have transactions T1={start_lsn=100, end_lsn=400} and
> T2={start_lsn=200, end_lsn=300}.
> Transaction T2 is sent to the replica and replica confirms that
> flush_lsn=300.
> If now we want to restart logical decoding, we can not start with
> position less than 300, because CreateDecodingContext doesn't allow it:
>
> * start_lsn
> *The LSN at which to start decoding. If InvalidXLogRecPtr, restart
> *from the slot's confirmed_flush; otherwise, start from the specified
> *location (but move it forwards to confirmed_flush if it's older than
> *that, see below).
> *
> else if (start_lsn < slot->data.confirmed_flush)
> {
> /*
> * It might seem like we should error out in this case, but it's
> * pretty common for a client to acknowledge a LSN it doesn't have to
> * do anything for, and thus didn't store persistently, because the
> * xlog records didn't result in anything relevant for logical
> * decoding. Clients have to be able to do that to support synchronous
> * replication.
> */
>
> So it means that we have no chance to restore T1?
> What is worse, if there are valid T2 transaction records with lsn >=
> 300, then we can partly decode T1 and send this T1' to the replica.
> I missed something here?

The decoding starts from restart_lsn of the slot, start_lsn is used for
skipping the transactions.

> Are there any alternative way to "seek" slot to the proper position
> without actual fetching data from it or recreation of the slot?

You can seek forward just fine, just specify the start position in
START_REPLICATION command.

> Is there any mechanism in xlog which can enforce consistent decoding of
> transaction (so that no transaction records are missed)?
> May be I missed something but I didn't find any "record_number" or
> something else which can identify first record of transaction.

As I mentioned above, what you probably want to do is use replication
origins. When you use those you get origin info when decoding the
transaction which you can then send to downstream and it can update it's
idea of where it is for that origin. This is especially useful for the
transaction forwarding you are doing (See BDR and/or pglogical code for
example of that).

--
Petr Jelinek http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services