Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

On Wed, Jun 10, 2020 at 5:11 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
>
> On Wed, Jun 10, 2020 at 5:02 PM Dilip Kumar <dilipbalaut(at)gmail(dot)com> wrote:
> >
> > On Wed, Jun 10, 2020 at 4:00 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> > >
> > > > 2. Files should not be closed at the end of the transaction:
> > > > Currently, files opened with BufFileCreateShared/BufFileOpenShared are
> > > > registered to be closed on EOXACT. Basically, we need to open the
> > > > changes file on the stream start and keep it open until stream stop,
> > > > so we can not afford to get it closed on the EOXACT. I have added a
> > > > flag for the same.
> > > >
> > >
> > > But where do we end the transaction before the stream stop which can
> > > lead to closure of this file?
> >
> > Currently, I am keeping the transaction only while creating/opening
> > the files and closing immediately after that, maybe we can keep the
> > transaction until stream stop, then we can avoid this changes, and we
> > can also avoid creating extra resource owner? What is your thought on
> > this?
> >
>
> I would prefer to keep the transaction until the stream stop unless
> there are good reasons for not doing so.

I am ready with the first patch set which replaces the temp file usage
in the worker with the buffile usage. (patch v27-0013 and v27-0014)

Open item:
- As of now, I have kept the buffile changes and the worker using
buffile as separate patches for review. Later I will make buffile
changes patch as a base patch and I will merge the worker changes with
the 0008 patch.

- Currently, while reading/writing the streaming/subxact files we are
reporting the wait event for example
'pgstat_report_wait_start(WAIT_EVENT_LOGICAL_SUBXACT_WRITE);', but
BufFileWrite/BufFileRead internally reports the read/write wait event.
So I think we can avoid reporting that? Basically, this part is still
I have to work upon, once we get the consensus then I can remove those
extra wait event from the patch.

- There are still a few open comments, from your other mails, I still
have to work upon. So I will work on those in the next version.

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com