Re: "stuck spinlock"

On Dec 12, 2013, at 6:24 PM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
> Is it really a regular pattern like hourly? What's your
> checkpoint_segments?

No, it's not a pattern like that; that's an approximation. Sometimes, they come in clusters, sometimes, 2-3 hours past without one. They don't happen exclusively inside or outside of a checkpoint.

checkpoint_timeout = 5min
checkpoint_segments = 64
checkpoint_completion_target = 0.9

> Could you, arround the time of a crash, check "grep Dirt
> /proc/meminfo" and run iostat -xm 1 20?

Dirty: 30104 kB

avg-cpu: %user %nice %system %iowait %steal %idle
3.70 0.00 0.91 0.53 0.00 94.85

Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.83 113.13 1.18 2.01 0.06 0.45 329.29 0.04 12.18 1.28 18.55 0.57 0.18
sdb 0.06 113.15 0.98 1.99 0.06 0.45 349.36 0.24 79.30 3.57 116.60 1.46 0.43
md0 0.00 0.00 0.00 0.00 0.00 0.00 3.39 0.00 0.00 0.00 0.00 0.00 0.00
md1 0.00 0.00 1.18 114.92 0.01 0.45 8.01 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.06 111.82 0.00 0.44 8.02 0.57 4.88 0.24 4.89 0.04 0.43
dm-1 0.00 0.00 1.11 3.03 0.00 0.01 8.00 1.25 300.47 0.38 410.89 0.17 0.07
sdc 0.00 0.00 12.10 136.13 0.50 19.97 282.85 1.94 13.07 2.30 14.03 0.55 8.20
dm-2 0.00 39.63 24.23 272.24 1.00 39.82 281.97 1.31 4.44 1.98 4.65 0.44 13.03
sdd 0.00 0.00 12.13 136.11 0.50 19.84 281.10 1.35 9.10 1.64 9.77 0.42 6.21

avg-cpu: %user %nice %system %iowait %steal %idle
1.09 0.00 0.08 0.13 0.00 98.71

Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 0.00 558.00 0.00 8.95 32.85 7.36 13.20 0.00 13.20 0.12 6.80
dm-2 0.00 28.00 0.00 558.00 0.00 8.95 32.85 7.38 13.23 0.00 13.23 0.12 6.80
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

avg-cpu: %user %nice %system %iowait %steal %idle
0.38 0.00 0.17 0.13 0.00 99.33

Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 36.00 11.00 0.18 0.15 14.30 0.06 1.36 0.67 3.64 0.94 4.40
dm-2 0.00 0.00 36.00 11.00 0.18 0.15 14.30 0.06 1.36 0.67 3.64 0.94 4.40
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

avg-cpu: %user %nice %system %iowait %steal %idle
0.83 0.00 0.29 0.04 0.00 98.83

Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 33.00 0.00 0.32 0.00 19.88 0.12 3.52 3.52 0.00 0.61 2.00
dm-2 0.00 0.00 33.00 0.00 0.32 0.00 19.88 0.12 3.52 3.52 0.00 0.61 2.00
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

avg-cpu: %user %nice %system %iowait %steal %idle
1.08 0.00 0.29 0.00 0.00 98.62

Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 6.00 4.00 0.07 0.03 20.80 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 6.00 4.00 0.07 0.03 20.80 0.00 0.00 0.00 0.00 0.00 0.00
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

avg-cpu: %user %nice %system %iowait %steal %idle
0.50 0.00 0.08 0.04 0.00 99.37

Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
md1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdc 0.00 0.00 11.00 3.00 0.22 0.12 49.14 0.00 0.00 0.00 0.00 0.00 0.00
dm-2 0.00 0.00 11.00 3.00 0.22 0.12 49.14 0.00 0.00 0.00 0.00 0.00 0.00
sdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

--
-- Christophe Pettus
xof(at)thebuild(dot)com