Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

On Tue, Apr 21, 2020 at 5:30 PM Dilip Kumar <dilipbalaut(at)gmail(dot)com> wrote:
>
> On Sat, Apr 18, 2020 at 6:12 PM Erik Rijkers <er(at)xs4all(dot)nl> wrote:
> >
> > On 2020-04-18 11:10, Erik Rijkers wrote:
> > > On 2020-04-18 11:07, Erik Rijkers wrote:
> > >>>> Hi Erik,
> > >>>>
> > >>>> While setting up the cascading replication I have hit one issue on
> > >>>> base code[1]. After fixing that I have got one crash with streaming
> > >>>> on patch. I am not sure whether you are facing any of these 2
> > >>>> issues
> > >>>> or any other issue. If your issue is not any of these then plese
> > >>>> share the callstack and steps to reproduce.
> > >>
> > >> I figured out a few things about this. Attached is a bash script
> > >> test.sh, to reproduce:
> > >
> > > And the attached file, test.sh. (sorry)
> >
> > It turns out I must have been mistaken somewhere. I probably missed
> > bugfix_in_schema_sent.patch)
> >
> > I have just now rebuilt all the instances on top of master with these
> > patches:
> >
> > > [v14-0001-Immediately-WAL-log-assignments.patch]
> > > [v14-0002-Issue-individual-invalidations-with.patch]
> > > [v14-0003-Extend-the-output-plugin-API-with-stream-methods.patch]
> > > [v14-0004-Gracefully-handle-concurrent-aborts-of-uncommitt.patch]
> > > [v14-0005-Implement-streaming-mode-in-ReorderBuffer.patch]
> > > [v14-0006-Add-support-for-streaming-to-built-in-replicatio.patch]
> > > [v14-0007-Track-statistics-for-streaming.patch]
> > > [v14-0008-Enable-streaming-for-all-subscription-TAP-tests.patch]
> > > [v14-0009-Add-TAP-test-for-streaming-vs.-DDL.patch]
> > > [v14-0010-Bugfix-handling-of-incomplete-toast-tuple.patch]
> > > [bugfix_in_schema_sent.patch]
> >
> > (by the way: this build's regression tests 'ddl', 'toast', and
> > 'spill' fail)
>
> Yeah, this is a. known issue, actually, while streaming the
> transaction the output message is changed. I have a plan to work on
> this part.

I have fixed this part. Basically, now, I have created a separate
function to get the streaming changes
'pg_logical_slot_get_streaming_changes'. So the default function
pg_logical_slot_get_changes will work as it is and test decoding test
cases will not fail.

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com