Re: Deadlock in XLogInsert at AIX

On 01/31/2017 05:03 PM, Konstantin Knizhnik wrote:
> One more assertion failure:
>
>
> ExceptionalCondition(conditionName = "!(OldPageRqstPtr <=
> XLogCtl->InitializedUpTo)", errorType = "FailedAssertion", fileName =
> "xlog.c", lineNumber = 1887), line 54 in "assert.c"
>
> (dbx) p OldPageRqstPtr
> 153551667200
> (dbx) p XLogCtl->InitializedUpTo
> 153551667200
> (dbx) p InitializedUpTo
> 153551659008
>
> I slightly modify xlog.c code - store value of XLogCtl->InitializedUpTo
> in local variable:
>
>
> 1870 LWLockAcquire(WALBufMappingLock, LW_EXCLUSIVE);
> 1871
> 1872 /*
> 1873 * Now that we have the lock, check if someone
> initialized the page
> 1874 * already.
> 1875 */
> 1876 while (upto >= XLogCtl->InitializedUpTo || opportunistic)
> 1877 {
> 1878 XLogRecPtr InitializedUpTo =
> XLogCtl->InitializedUpTo;
> 1879 nextidx = XLogRecPtrToBufIdx(InitializedUpTo);
> 1880
> 1881 /*
> 1882 * Get ending-offset of the buffer page we need
> to replace (this may
> 1883 * be zero if the buffer hasn't been used yet).
> Fall through if it's
> 1884 * already written out.
> 1885 */
> 1886 OldPageRqstPtr = XLogCtl->xlblocks[nextidx];
> 1887 Assert(OldPageRqstPtr <= XLogCtl->InitializedUpTo);
>
>
> And, as you can see, XLogCtl->InitializedUpTo is not equal to saved
> value InitializedUpTo.
> But we are under exclusive WALBufMappingLock and InitializedUpTo is
> updated only under this lock.
> So it means that LW-locks doesn't work!

Yeah, so it seems. XLogCtl->InitializeUpTo is quite clearly protected by
the WALBufMappingLock. All references to it (after StartupXLog) happen
while holding the lock.

Can you get the assembly output of the AdvanceXLInsertBuffer() function?
I wonder if the compiler is rearranging things so that
XLogCtl->InitializedUpTo is fetched before the LWLockAcquire call. Or
should there be a memory barrier instruction somewhere in LWLockAcquire?

- Heikki