Re: Connections hang indefinitely while taking a gin index's LWLock buffer_content lock

On Mon, Oct 29, 2018 at 12:04 PM chenhj <chjischj(at)163(dot)com> wrote:
> #4 0x00000000007095ac in LWLockAcquire (lock=0x2aaab4009564, mode=LW_EXCLUSIVE) at lwlock.c:1233
> #5 0x0000000000490a32 in ginStepRight (buffer=6713826, index=<optimized out>, lockmode=lockmode(at)entry=2) at ginbtree.c:174
> #6 0x0000000000490c1c in ginFinishSplit (btree=btree(at)entry=0x7ffd81e4f950, stack=0x28eebf8, freestack=freestack(at)entry=1 '\001', buildStats=buildStats(at)entry=0x0) at ginbtree.c:701
> #7 0x0000000000491396 in ginInsertValue (btree=btree(at)entry=0x7ffd81e4f950, stack=<optimized out>, insertdata=insertdata(at)entry=0x7ffd81e4f940, buildStats=buildStats(at)entry=0x0)
> at ginbtree.c:773
> #8 0x000000000048fb42 in ginInsertItemPointers (index=<optimized out>, rootBlkno=rootBlkno(at)entry=644, items=items(at)entry=0x2916598, nitem=nitem(at)entry=353, buildStats=buildStats(at)entry=0x0)
> at gindatapage.c:1907
> #9 0x000000000048a9ea in ginEntryInsert (ginstate=ginstate(at)entry=0x28e7ef8, attnum=<optimized out>, key=42920440, category=<optimized out>, items=0x2916598, nitem=353,
> buildStats=buildStats(at)entry=0x0) at gininsert.c:214
> #10 0x0000000000496d94 in ginInsertCleanup (ginstate=ginstate(at)entry=0x28e7ef8, full_clean=full_clean(at)entry=0 '\000', fill_fsm=fill_fsm(at)entry=1 '\001',
> forceCleanup=forceCleanup(at)entry=0 '\000', stats=stats(at)entry=0x0) at ginfast.c:883
> #11 0x0000000000497727 in ginHeapTupleFastInsert (ginstate=ginstate(at)entry=0x28e7ef8, collector=collector(at)entry=0x7ffd81e4fe60) at ginfast.c:448
> #12 0x000000000048b209 in gininsert (index=<optimized out>, values=0x7ffd81e4ff40, isnull=0x7ffd81e50040 "", ht_ctid=0x280d98c, heapRel=<optimized out>, checkUnique=<optimized out>,
> indexInfo=0x28b5aa8) at gininsert.c:522

I want to point something out about the above inserter represented by
this stack trace. It deadlocks against VACUUM from within the
following ginEntryInsert() call:

/*
* While we left the page unlocked, more stuff might have gotten
* added to it. If so, process those entries immediately. There
* shouldn't be very many, so we don't worry about the fact that
* we're doing this with exclusive lock. Insertion algorithm
* guarantees that inserted row(s) will not continue on next page.
* NOTE: intentionally no vacuum_delay_point in this loop.
*/
if (PageGetMaxOffsetNumber(page) != maxoff)
{
ginInitBA(&accum);
processPendingPage(&accum, &datums, page, maxoff + 1);

ginBeginBAScan(&accum);
while ((list = ginGetBAEntry(&accum,
&attnum, &key, &category,
&nlist)) != NULL)
ginEntryInsert(ginstate, attnum, key, category, //
<--- deadlocks
list, nlist, NULL);
}

Clearly this particular call to ginEntryInsert() from within
ginInsertCleanup() is generally supposed to be a rare occurrence. It's
not surprising that it's hard for you to hit this issue.

BTW, I strongly doubt that disabling hugepages has fixed anything,
though it may have accidentally masked the problem. This is an issue
around the basic correctness of the locking protocols used by GIN.

--
Peter Geoghegan