Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

On Fri, Nov 15, 2019 at 4:01 PM Dilip Kumar <dilipbalaut(at)gmail(dot)com> wrote:
>
> On Fri, Nov 15, 2019 at 3:50 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> >
> >
> > Few other comments on this patch:
> > 1.
> > + case REORDER_BUFFER_CHANGE_INVALIDATION:
> > +
> > + /*
> > + * Execute the invalidation message locally.
> > + *
> > + * XXX Do we need to care about relcacheInitFileInval and
> > + * the other fields added to ReorderBufferChange, or just
> > + * about the message itself?
> > + */
> > + LocalExecuteInvalidationMessage(&change->data.inval.msg);
> > + break;
> >
> > Here, why are we executing messages individually? Can't we just
> > follow what we do in DecodeCommit which is to record the invalidations
> > in ReorderBufferTXN as we encounter them and then allow them to
> > execute on each REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID. Is there a
> > reason why we don't do ReorderBufferXidSetCatalogChanges when we
> > receive any invalidation message?
> IMHO, the reason is that in DecodeCommit, we get all the invalidation
> at one time so, at REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID, we don't
> know which invalidation message to execute so for being safe we have
> to execute all. But, since we are logging all invalidation
> individually, we exactly know at this stage which cache to invalidate.
> So it is better to only invalidate required cache not all.
>

In that case, invalidations can be processed multiple times, the first
time when these individual WAL logs for invalidation are processed and
then later at commit time when we accumulate all invalidation messages
and then execute them for REORDER_BUFFER_CHANGE_INTERNAL_COMMAND_ID.
Can we avoid to execute invalidations from other places after this
patch which also includes executing them as part of XLOG_INVALIDATIONS
processing?

--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com