Re: An example of bugs for Hot Standby

On Wed, 2009-12-16 at 14:05 +0000, Simon Riggs wrote:
> On Wed, 2009-12-16 at 10:33 +0000, Simon Riggs wrote:
> > On Tue, 2009-12-15 at 20:25 +0900, Hiroyuki Yamada wrote:
> > > Hot Standby node can freeze when startup process calls LockBufferForCleanup().
> > > This bug can be reproduced by the following procedure.
> >
> > Interesting. Looks like this can happen, which is a shame cos I just
> > removed the wait checking code after not ever having seen a wait.
> >
> > Thanks for the report.
> >
> > Must-fix item for HS.
>
> So this deadlock can happen at two places:
>
> 1. When a relation lock waits behind an AccessExclusiveLock and then
> Startup runs LockBufferForCleanup()
>
> 2. When Startup is a pin count waiter and a lock acquire begins to wait
> on a relation lock
>
> So we must put in direct deadlock detection in both places. We can't use
> the normal deadlock detector because in case (1) the backend might
> already have exceeded deadlock_timeout.
>
> Proposal:

Better proposal

* It's possible for 3-way deadlocks to occur in Hot Standby mode.
* If a user backend sleeps on a lock while it holds a buffer pin that
* leaves open the risk of deadlock. The user backend will only sleep
* if it waits behind an AccessExclusiveLock held by Startup process.
* If the Startup process then tries to access any buffer that is pinned
* then it too will sleep and neither process will ever wake.
*
* We need to make a deadlock check in two places: in the user backend
* when we sleep on a lock, and in the Startup process when we sleep
* on a buffer pin. We need both checks because the deadlock can occur
* from both directions.
*
* Just before a user backend sleeps on a lock, we accumulate a list of
* buffers pinned by the backend. We then grab the an LWlock
* and then check each of the buffers to see if the Startup process is
* waiting on them. If so, we release the lock and throw deadlock error.
* If Startup process is not waiting we then record the pinned buffers
* in the BufferDeadlockRisk data structure and release the lock.
* When we later get the lock we remove the deadlock risk.
*
* When the Startup process is about to wait on a buffer pin it checks
* the buffer it is about to pin in the BufferDeadlockRisk list. If the
* buffer is already held by one or more lock waiters then we send a
* conflict cancel to them and wait for them to die before rechecking
* the buffer lock.

This way we only cancel direct deadlocks.

It doesn't solve general problem of buffer waits, but they may be
solvable by different mechanism.

--
Simon Riggs www.2ndQuadrant.com