RE: Rework LogicalOutputPluginWriterUpdateProgress

On Sun, Feb 19, 2023 at 21:06 PM Wang, Wei/王 威 <wangw(dot)fnst(at)fujitsu(dot)com> wrote:
> On Thur, Feb 14, 2023 at 2:03 AM Andres Freund <andres(at)anarazel(dot)de> wrote:
> > On 2023-02-13 14:06:57 +0530, Amit Kapila wrote:
> > > > > The patch calls update_progress in change_cb_wrapper and other
> > > > > wrappers which will miss the case of DDLs that generates a lot of data
> > > > > that is not processed by the plugin. I think for that we either need
> > > > > to call update_progress from reorderbuffer.c similar to what the patch
> > > > > has removed or we need some other way to address it. Do you have any
> > > > > better idea?
> > > >
> > > > I don't mind calling something like update_progress() in the specific cases
> > > > that's needed, but I think those are just the
> > > > if (!RelationIsLogicallyLogged(relation))
> > > > if (relation->rd_rel->relrewrite && !rb->output_rewrites))
> > > >
> > > > To me it makes a lot more sense to call update_progress() for those, rather
> > > > than generally.
> > > >
> > >
> > > Won't it be better to call it wherever we don't invoke any wrapper
> > > function like for cases REORDER_BUFFER_CHANGE_INVALIDATION, sequence
> > > changes, etc.? I was thinking that wherever we don't call the wrapper
> > > function which means we don't have a chance to invoke
> > > update_progress(), the timeout can happen if there are a lot of such
> > > messages.
> >
> > ISTM that the likelihood of causing harm due to increased overhead is higher
> > than the gain.
>
> I would like to do something for this thread. So, I am planning to update the
> patch as per discussion in the email chain unless someone is already working on
> it.

Thanks to Andres and Amit for the discussion.

Based on the discussion and Andres' WIP(in [1]), I made the following
modifications:
1. Some function renaming stuffs.
2. Added the threshold-related logic in the function
update_progress_and_keepalive.
3. Added the timeout-related processing of temporary data and
unlogged/foreign/system tables in the function ReorderBufferProcessTXN.
4. Improved error messages in the function OutputPluginPrepareWrite.
5. Invoked function update_progress_and_keepalive to fix sync-related problems
caused by filters such as origin in functions DecodeCommit(), DecodePrepare()
and ReorderBufferAbort();
6. Removed the invocation of function update_progress_and_keepalive in the
function begin_prepare_cb_wrapper().
7. Invoked the function update_progress_and_keepalive() in the function
stream_truncate_cb_wrapper(), just like we do in the function
truncate_cb_wrapper().
8. Removed the check of SyncRepRequested() in the syncrep logic in the function
WalSndUpdateProgressAndKeepAlive();
9. Added the check for wal_sender_timeout before using it in functions
WalSndUpdateProgressAndKeepAlive() and WalSndWriteData();

Attach the new patch.

[1] - https://www.postgresql.org/message-id/20230208200235.esfoggsmuvf4pugt%40awork3.anarazel.de

Regards,
Wang wei