Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

On Mon, Jun 22, 2020 at 4:30 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
>
> On Mon, Jun 22, 2020 at 4:26 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> >
> > On Thu, Jun 18, 2020 at 9:02 PM Dilip Kumar <dilipbalaut(at)gmail(dot)com> wrote:
> > >
> > > Yes, I have made the changes. Basically, now I am only using the
> > > XLOG_XACT_INVALIDATIONS for generating all the invalidation messages.
> > > So whenever we are getting the new set of XLOG_XACT_INVALIDATIONS, we
> > > are directly appending it to the txn->invalidations. I have tested
> > > the XLOG_INVALIDATIONS part but while sending this mail I realized
> > > that we could write some automated test for the same.
> > >
> >
> > Can you share how you have tested it?
> >
> > > I will work on
> > > that soon.
> > >
> >
> > Cool, I think having a regression test for this will be a good idea.
> >
>
> Other than above tests, can we somehow verify that the invalidations
> generated at commit time are the same as what we do with this patch?
> We have verified with individual commands but it would be great if we
> can verify for the regression tests.

I have verified this using a few random test cases. For verifying
this I have made some temporary code changes with an assert as shown
below. Basically, on DecodeCommit we call
ReorderBufferAddInvalidations function only for an assert checking.

-void
ReorderBufferAddInvalidations(ReorderBuffer *rb, TransactionId xid,
XLogRecPtr
lsn, Size nmsgs,
-
SharedInvalidationMessage *msgs)
+
SharedInvalidationMessage *msgs, bool commit)
{
ReorderBufferTXN *txn;

txn = ReorderBufferTXNByXid(rb, xid, true, NULL, lsn, true);
-
+ if (commit)
+ {
+ Assert(txn->ninvalidations == nmsgs);
+ return;
+ }

The result is that for a normal local test it works fine. But with
regression suit, it hit an assert at many places because if the
rollback of the subtransaction is involved then at commit time
invalidation messages those are not logged whereas with command time
invalidation those are logged.

As of now, I have only put assert on the count, if we need to verify
the exact messages then we might need to somehow categories the
invalidation messages because the ordering of the messages will not be
the same. For testing this we will have to arrange them by category
i.e relcahce, catcache and then we can compare them.

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com