Re: hung backends stuck in spinlock heavy endless loop

On Tue, Jan 13, 2015 at 5:21 PM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
> On 2015-01-13 15:17:15 -0800, Peter Geoghegan wrote:
>> I'm inclined to think that this is a livelock, and so the problem
>> isn't evident from the structure of the B-Tree, but it can't hurt to
>> check.
>
> My guess is rather that it's contention on the freelist lock via
> StrategyGetBuffer's. I've seen profiles like this due to exactly that
> before - and it fits to parallel loading quite well.

I think I've got it to pop again. s_lock is only showing 35%
(increasing very slowly if at all) but performance is mostly halted.
Frame pointer is compiled out. perf report attached.

merlin

35.82% postgres [.] s_lock
23.71% postgres [.] tas
14.01% postgres [.] tas
6.82% postgres [.] spin_delay
5.93% postgres [.] LWLockRelease
4.36% postgres [.] LWLockAcquireCommon
2.34% postgres [.] FunctionCall2Coll
1.79% postgres [.] _bt_compare
0.80% postgres [.] LockBuffer
0.52% postgres [.] btoidcmp
0.49% postgres [.] ReleaseAndReadBuffer
0.47% postgres [.] _bt_moveright
0.36% postgres [.] _bt_checkpage
0.36% postgres [.] _bt_relandgetbuf
0.20% perf [.] 0x000000000004312a
0.19% postgres [.] LWLockAcquire
0.13% libv8.so [.] 0x00000000001bbbe0
0.11% libc-2.17.so [.] 0x0000000000151134
0.09% libwebkit.so [.] 0x000000000106ccb7
0.05% libgdk_pixbuf-2.0.so.0.2800.1 [.] 0x00000000000139c7
0.04% Xorg [.] 0x00000000000efb00
0.03% libglib-2.0.so.0.3800.1 [.] 0x00000000000876a2
0.03% [kernel] [k] __ticket_spin_lock
0.02% perf-12966.map [.] 0x00000625db944621
0.02% libskia.so [.] 0x000000000021d15f
0.02% libcairo.so.2.11200.16 [.] 0x000000000002440b
0.02% libpthread-2.17.so [.]
__pthread_mutex_unlock_usercnt
0.02% [kernel] [k]
copy_user_enhanced_fast_string
0.02% radeon_drv.so [.] 0x0000000000045002
0.02% libpthread-2.17.so [.] pthread_mutex_lock
0.02% [kernel] [k] fget_light
0.01% [kernel] [k] __schedule
0.01% libexa.so [.] 0x0000000000006e1d
0.01% libgdk-x11-2.0.so.0.2400.20 [.] 0x000000000002f0b0
0.01% libvte.so.9.2800.2 [.] 0x0000000000039028
0.01% [radeon] [k] r100_mm_rreg
0.01% [kernel] [k] xhci_irq
0.01% [kernel] [k] native_write_msr_safe
0.01% [kernel] [k]
update_cfs_rq_blocked_load
0.01% libglib-2.0.so.0.3800.1 [.] g_hash_table_lookup
0.01% libgobject-2.0.so.0.3800.1 [.]
g_type_check_instance_is_a
Press '?' for help on key bindings