Re: Exit walsender before confirming remote flush in logical replication

On Fri, Jan 20, 2023 at 4:15 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
>
> On Tue, Jan 17, 2023 at 2:41 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> >
> > Let me try to summarize the discussion till now. The problem we are
> > trying to solve here is to allow a shutdown to complete when walsender
> > is not able to send the entire WAL. Currently, in such cases, the
> > shutdown fails. As per our current understanding, this can happen when
> > (a) walreceiver/walapply process is stuck (not able to receive more
> > WAL) due to locks or some other reason; (b) a long time delay has been
> > configured to apply the WAL (we don't yet have such a feature for
> > logical replication but the discussion for same is in progress).
> >
> > Both reasons mostly apply to logical replication because there is no
> > separate walreceiver process whose job is to just flush the WAL. In
> > logical replication, the process that receives the WAL also applies
> > it. So, while applying it can stuck for a long time waiting for some
> > heavy-weight lock to be released by some other long-running
> > transaction by the backend.
> >
>
> While checking the commits and email discussions in this area, I came
> across the email [1] from Michael where something similar seems to
> have been discussed. Basically, whether the early shutdown of
> walsender can prevent a switchover between publisher and subscriber
> but that part was never clearly answered in that email chain. It might
> be worth reading the entire discussion [2]. That discussion finally
> lead to the following commit:

Right, in the thread the question is raised about whether it makes
sense for logical replication to send all WALs but there is no
conclusion on that. But I think this patch is mainly about resolving
the PANIC due to extra WAL getting generated by walsender during
checkpoint processing and that's the reason the behavior of sending
all the WAL is maintained but only the extra WAL generation stopped
(before shutdown checkpoint can proceed) using this new state

>
> commit c6c333436491a292d56044ed6e167e2bdee015a2
> Author: Andres Freund <andres(at)anarazel(dot)de>
> Date: Mon Jun 5 18:53:41 2017 -0700
>
> Prevent possibility of panics during shutdown checkpoint.
>
> When the checkpointer writes the shutdown checkpoint, it checks
> afterwards whether any WAL has been written since it started and
> throws a PANIC if so. At that point, only walsenders are still
> active, so one might think this could not happen, but walsenders can
> also generate WAL, for instance in BASE_BACKUP and logical decoding
> related commands (e.g. via hint bits). So they can trigger this panic
> if such a command is run while the shutdown checkpoint is being
> written.
>
> To fix this, divide the walsender shutdown into two phases. First,
> checkpointer, itself triggered by postmaster, sends a
> PROCSIG_WALSND_INIT_STOPPING signal to all walsenders. If the backend
> is idle or runs an SQL query this causes the backend to shutdown, if
> logical replication is in progress all existing WAL records are
> processed followed by a shutdown.
> ...
> ...
>
> Here, as mentioned in the commit, we are trying to ensure that before
> checkpoint writes its shutdown WAL record, we ensure that "if logical
> replication is in progress all existing WAL records are processed
> followed by a shutdown.". I think even before this commit, we try to
> send the entire WAL before shutdown but not completely sure.

Yes, I think that there is no change in that behavior.

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com