Re: Skipping logical replication transactions on subscriber side

On Wed, Jan 26, 2022 at 12:54 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
>
> On Wed, Jan 26, 2022 at 8:55 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> >
> > On Wed, Jan 26, 2022 at 11:51 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> > >
> > > On Wed, Jan 26, 2022 at 11:28 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> > > >
> > > > IIUC, the proposal is to compare the skip_xid with the very
> > > > transaction the apply worker received to apply and raise a warning if
> > > > it doesn't match with skip_xid and then continue. This seems like a
> > > > reasonable idea but can we guarantee that it is always the first
> > > > transaction that we want to skip? We seem to guarantee that we won't
> > > > get something again once it is written durably/flushed on the
> > > > subscriber side. I guess here it can happen that before the errored
> > > > transaction, there is some empty xact, or maybe part of the stream
> > > > (consider streaming transactions) of some xact, or there could be
> > > > other cases as well where the server will send those xacts again.
> > >
> > > Good point.
> > >
> > > I guess that in the situation the worker entered an error loop, we can
> > > guarantee that the worker fails while applying the first non-empty
> > > transaction since starting logical replication. And the transaction is
> > > what we’d like to skip. If the transaction that can be applied without
> > > an error is resent after a restart, it’s a problem of logical
> > > replication. As you pointed out, it's possible that there are some
> > > empty transactions before the transaction in question since we don't
> > > advance replication origin LSN if the transaction is empty. Also,
> > > probably the same is true for a streamed transaction that is rolled
> > > back or ROLLBACK-PREPARED transactions. So, we can also skip clearing
> > > subskipxid if the transaction is empty? That is, we make sure to clear
> > > it after applying the first non-empty transaction. We would need to
> > > carefully think about this solution otherwise ALTER SUBSCRIPTION SKIP
> > > ends up not working at all in some cases.
>
> I think it is okay to clear after the first successful application of
> any transaction. What I was not sure was about the idea of giving
> WARNING/ERROR if the first xact to be applied is not the same as
> skip_xid.

Do you prefer not to do anything in this case?

>
> >
> > Probably, we also need to consider the case where the tablesync worker
> > entered an error loop and the user wants to skip the transaction? The
> > apply worker is also running at the same time but it should not clear
> > subskipxid. Similarly, the tablesync worker should not clear
> > subskipxid if the apply worker wants to skip the transaction.
> >
>
> I think for tablesync workers, the skip_xid set via this mechanism
> won't work as we don't have any remote_xid for them, and neither any
> XID is reported in the view for them.

If the tablesync worker raises an error while applying changes after
finishing the copy, it also reports the error XID.

Regards,

--
Masahiko Sawada
EDB: https://www.enterprisedb.com/