Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

On Fri, Jan 10, 2020 at 10:53 AM Dilip Kumar <dilipbalaut(at)gmail(dot)com> wrote:
>
> On Mon, Dec 30, 2019 at 3:43 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> >
> >
> > > 2. During commit time in DecodeCommit we check whether we need to skip
> > > the changes of the transaction or not by calling
> > > SnapBuildXactNeedsSkip but since now we support streaming so it's
> > > possible that before we decode the commit WAL, we might have already
> > > sent the changes to the output plugin even though we could have
> > > skipped those changes. So my question is instead of checking at the
> > > commit time can't we check before adding to ReorderBuffer itself
> > >
> >
> > I think if we can do that then the same will be true for current code
> > irrespective of this patch. I think it is possible that we can't take
> > that decision while decoding because we haven't assembled a consistent
> > snapshot yet. I think we might be able to do that while we try to
> > stream the changes. I think we need to take care of all the
> > conditions during streaming (when the logical_decoding_workmem limit
> > is reached) as we do in DecodeCommit. This needs a bit more study.
>
> I have analyzed this further and I think we can not decide all the
> conditions even while streaming. Because IMHO once we get the
> SNAPBUILD_FULL_SNAPSHOT we can add the changes to the reorder buffer
> so that if we get the commit of the transaction after we reach to the
> SNAPBUILD_CONSISTENT. However, if we get the commit before we reach
> to SNAPBUILD_CONSISTENT then we need to ignore this transaction. Now,
> even if we have SNAPBUILD_FULL_SNAPSHOT we can stream the changes
> which might get dropped later but that we can not decide while
> streaming.
>

This makes sense to me, but we should add a comment for the same when
we are streaming to say we can't skip similar to how we do during
commit time because of the above reason described by you. Also, what
about other conditions where we can skip the transaction, basically
cases like (a) when the transaction happened in another database, (b)
when the output plugin is not interested in the origin and (c) when we
are doing fast-forwarding

--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com