Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

On Wed, Jan 5, 2022 at 9:46 AM Andres Freund <andres(at)anarazel(dot)de> wrote:

> Hi,
>
> On 2021-12-29 11:31:51 -0800, Andres Freund wrote:
> > That's pretty much the same - XLogInsert() can trigger an
> > XLogWrite()/Flush().
> >
> > I think it's a complete no-go to add throttling to these places. It's
> quite
> > possible that it'd cause new deadlocks, and it's almost guaranteed to
> have
> > unintended consequences (e.g. replication falling back further because
> > XLogFlush() is being throttled).
>
> I thought of another way to implement this feature. What if we checked the
> current distance somewhere within XLogInsert(), but only set
> InterruptPending=true, XLogDelayPending=true. Then in ProcessInterrupts()
> we
> check if XLogDelayPending is true and sleep the appropriate time.
>
> That way the sleep doesn't happen with important locks held / within a
> critical section, but we still delay close to where we went over the
> maximum
> lag. And the overhead should be fairly minimal.
>

+1 to the idea, this way we can fairly throttle large and
smaller transactions the same way. I will work on this model and share the
patch. Please note that the lock contention still exists in this case.

> I'm doubtful that implementing the waits on a transactional level provides
> a
> meaningful enough amount of control - there's just too much WAL that can be
> generated within a transaction.
>

>
> Greetings,
>
> Andres Freund
>