Re: hung backends stuck in spinlock heavy endless loop

On Wed, Jan 14, 2015 at 9:30 AM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
> If you gdb in, and type 'fin' a couple times, to wait till the function
> finishes, is there actually any progress? I'm wondering whether it's
> just many catalog accesses + contention, or some other
> problem. Alternatively set a breakpoint on ScanPgRelation() or so and
> see how often it's hit.

well, i restarted the database, forgetting my looper was running which
immediately spun up and it got stuck again with a similar profile
(lots of cpu in spinlock):

Samples: 3K of event 'cycles', Event count (approx.): 2695723228
31.16% postgres [.] s_lock
22.32% postgres [.] tas
12.13% postgres [.] tas
5.93% postgres [.] spin_delay
5.69% postgres [.] LWLockRelease
3.75% postgres [.] LWLockAcquireCommon
3.61% perf [.] 0x00000000000526c4
2.51% postgres [.] FunctionCall2Coll
1.48% libc-2.17.so [.] 0x000000000016a132

> If you gdb in, and type 'fin' a couple times,
(gdb) fin
Run till exit from #0 0x00007ff4c63f7a97 in semop () from
/lib/x86_64-linux-gnu/libc.so.6
0x00000000006de073 in PGSemaphoreLock ()
(gdb) fin
Run till exit from #0 0x00000000006de073 in PGSemaphoreLock ()

It returned once. Second time, it didn't at least so far (minute or so).

>> I can send the code off-list if you guys think it'd help.
>
> Might be interesting.

sent (off-list).

merlin