Re: Really bad diskio

At Fri, 15 Jul 2005 14:00:07 -0700,
Jeffrey W. Baker wrote:
>
> On Fri, 2005-07-15 at 14:39 -0600, Ron Wills wrote:
> > Hello all
> >
> > I'm running a postgres 7.4.5, on a dual 2.4Ghz Athlon, 1Gig RAM and
> > an 3Ware SATA raid. Currently the database is only 16G with about 2
> > tables with 500000+ row, one table 200000+ row and a few small
> > tables. The larger tables get updated about every two hours. The
> > problem I having with this server (which is in production) is the disk
> > IO. On the larger tables I'm getting disk IO wait averages of
> > ~70-90%. I've been tweaking the linux kernel as specified in the
> > PostgreSQL documentations and switched to the deadline
> > scheduler. Nothing seems to be fixing this. The queries are as
> > optimized as I can get them. fsync is off in an attempt to help
> > preformance still nothing. Are there any setting I should be look at
> > the could improve on this???
>
> Can you please characterize this a bit better? Send the output of
> vmstat or iostat over several minutes, or similar diagnostic
> information.
>
> Also please describe your hardware more.

Here's a bit of a dump of the system that should be useful.

Processors x2:

vendor_id : AuthenticAMD
cpu family : 6
model : 8
model name : AMD Athlon(tm) MP 2400+
stepping : 1
cpu MHz : 2000.474
cache size : 256 KB

MemTotal: 903804 kB

Mandrake 10.0 Linux kernel 2.6.3-19mdk

The raid controller, which is using the hardware raid configuration:

3ware 9000 Storage Controller device driver for Linux v2.26.02.001.
scsi0 : 3ware 9000 Storage Controller
3w-9xxx: scsi0: Found a 3ware 9000 Storage Controller at 0xe8020000, IRQ: 17.
3w-9xxx: scsi0: Firmware FE9X 2.02.00.011, BIOS BE9X 2.02.01.037, Ports: 4.
Vendor: 3ware Model: Logical Disk 00 Rev: 1.00
Type: Direct-Access ANSI SCSI revision: 00
SCSI device sda: 624955392 512-byte hdwr sectors (319977 MB)
SCSI device sda: drive cache: write back, no read (daft)

This is also on a 3.6 reiser filesystem.

Here's the iostat for 10mins every 10secs. I've removed the stats from
the idle drives to reduce the size of this email.

Linux 2.6.3-19mdksmp (photo_server) 07/15/2005

avg-cpu: %user %nice %sys %iowait %idle
2.85 1.53 2.15 39.52 53.95

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 82.49 4501.73 188.38 1818836580 76110154

avg-cpu: %user %nice %sys %iowait %idle
0.30 0.00 1.00 96.30 2.40

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 87.80 6159.20 340.00 61592 3400

avg-cpu: %user %nice %sys %iowait %idle
2.50 0.00 1.45 94.35 1.70

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 89.60 5402.40 320.80 54024 3208

avg-cpu: %user %nice %sys %iowait %idle
1.00 0.10 1.35 97.55 0.00

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 105.20 5626.40 332.80 56264 3328

avg-cpu: %user %nice %sys %iowait %idle
0.40 0.00 1.00 87.40 11.20

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 92.61 4484.32 515.48 44888 5160

avg-cpu: %user %nice %sys %iowait %idle
0.45 0.00 1.00 92.66 5.89

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 89.10 4596.00 225.60 45960 2256

avg-cpu: %user %nice %sys %iowait %idle
0.30 0.00 0.80 96.30 2.60

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 86.49 3877.48 414.01 38736 4136

avg-cpu: %user %nice %sys %iowait %idle
0.50 0.00 1.00 98.15 0.35

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 97.10 4710.49 405.19 47152 4056

avg-cpu: %user %nice %sys %iowait %idle
0.35 0.00 1.00 98.65 0.00

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 93.30 5324.80 186.40 53248 1864

avg-cpu: %user %nice %sys %iowait %idle
0.40 0.00 1.10 96.70 1.80

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 117.88 5481.72 402.80 54872 4032

avg-cpu: %user %nice %sys %iowait %idle
0.50 0.00 1.05 98.30 0.15

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 124.00 6081.60 403.20 60816 4032

avg-cpu: %user %nice %sys %iowait %idle
8.75 0.00 2.55 84.46 4.25

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 125.20 5609.60 228.80 56096 2288

avg-cpu: %user %nice %sys %iowait %idle
2.25 0.00 1.30 96.00 0.45

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 176.98 6166.17 686.29 61600 6856

avg-cpu: %user %nice %sys %iowait %idle
5.95 0.00 2.25 88.09 3.70

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 154.55 7879.32 295.70 78872 2960

avg-cpu: %user %nice %sys %iowait %idle
10.29 0.00 3.40 81.97 4.35

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 213.19 11422.18 557.84 114336 5584

avg-cpu: %user %nice %sys %iowait %idle
1.90 0.10 3.25 94.75 0.00

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 227.80 12330.40 212.80 123304 2128

avg-cpu: %user %nice %sys %iowait %idle
0.55 0.00 0.85 96.80 1.80

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 96.30 3464.80 568.80 34648 5688

avg-cpu: %user %nice %sys %iowait %idle
0.70 0.00 1.10 97.25 0.95

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 92.60 4989.60 237.60 49896 2376

avg-cpu: %user %nice %sys %iowait %idle
2.75 0.00 2.10 93.55 1.60

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 198.40 10031.63 458.86 100216 4584

avg-cpu: %user %nice %sys %iowait %idle
0.65 0.00 2.40 95.90 1.05

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 250.25 14174.63 231.77 141888 2320

avg-cpu: %user %nice %sys %iowait %idle
0.60 0.00 2.15 97.20 0.05

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 285.50 12127.20 423.20 121272 4232

avg-cpu: %user %nice %sys %iowait %idle
0.60 0.00 2.90 95.65 0.85

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 393.70 14383.20 534.40 143832 5344

avg-cpu: %user %nice %sys %iowait %idle
0.55 0.00 2.15 96.15 1.15

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 252.15 11801.80 246.15 118136 2464

avg-cpu: %user %nice %sys %iowait %idle
0.75 0.00 3.45 95.15 0.65

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 396.00 19980.80 261.60 199808 2616

avg-cpu: %user %nice %sys %iowait %idle
0.70 0.00 2.70 95.70 0.90

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 286.20 14182.40 467.20 141824 4672

avg-cpu: %user %nice %sys %iowait %idle
0.70 0.00 2.70 95.65 0.95

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 344.20 15838.40 473.60 158384 4736

avg-cpu: %user %nice %sys %iowait %idle
0.75 0.00 1.70 97.50 0.05

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 178.72 7495.70 412.39 75032 4128

avg-cpu: %user %nice %sys %iowait %idle
1.05 0.05 1.30 97.05 0.55

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 107.89 4334.87 249.35 43392 2496

avg-cpu: %user %nice %sys %iowait %idle
0.55 0.00 1.30 98.10 0.05

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 107.01 6345.55 321.12 63392 3208

avg-cpu: %user %nice %sys %iowait %idle
0.65 0.00 1.05 97.55 0.75

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 107.79 3908.89 464.34 39128 4648

avg-cpu: %user %nice %sys %iowait %idle
0.50 0.00 1.15 97.75 0.60

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 109.21 4162.56 434.83 41584 4344

avg-cpu: %user %nice %sys %iowait %idle
0.75 0.00 1.15 98.00 0.10

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 104.19 4796.81 211.58 48064 2120

avg-cpu: %user %nice %sys %iowait %idle
0.70 0.00 1.05 97.85 0.40

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 105.50 4690.40 429.60 46904 4296

avg-cpu: %user %nice %sys %iowait %idle
0.75 0.00 1.10 98.15 0.00

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 107.51 4525.33 357.96 45208 3576

avg-cpu: %user %nice %sys %iowait %idle
2.80 0.00 1.65 92.81 2.75

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 123.18 3810.59 512.29 38144 5128

avg-cpu: %user %nice %sys %iowait %idle
0.60 0.00 1.05 97.10 1.25

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 104.60 3780.00 236.00 37800 2360

avg-cpu: %user %nice %sys %iowait %idle
0.70 0.00 1.10 95.96 2.25

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 117.08 3817.78 466.73 38216 4672

avg-cpu: %user %nice %sys %iowait %idle
0.65 0.00 0.90 96.65 1.80

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 117.20 3629.60 477.60 36296 4776

avg-cpu: %user %nice %sys %iowait %idle
0.80 0.00 1.10 97.50 0.60

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 112.79 4258.94 326.07 42632 3264

avg-cpu: %user %nice %sys %iowait %idle
1.05 0.15 1.20 97.50 0.10

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 125.83 2592.99 522.12 25904 5216

avg-cpu: %user %nice %sys %iowait %idle
0.60 0.00 0.55 98.20 0.65

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 104.90 823.98 305.29 8248 3056

avg-cpu: %user %nice %sys %iowait %idle
0.50 0.00 0.65 98.75 0.10

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 109.80 734.40 468.80 7344 4688

avg-cpu: %user %nice %sys %iowait %idle
1.15 0.00 1.05 97.75 0.05

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 107.70 751.20 463.20 7512 4632

avg-cpu: %user %nice %sys %iowait %idle
6.50 0.00 1.85 90.25 1.40

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 98.00 739.14 277.08 7384 2768

avg-cpu: %user %nice %sys %iowait %idle
0.20 0.00 0.40 82.75 16.65

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 83.13 550.90 360.08 5520 3608

avg-cpu: %user %nice %sys %iowait %idle
2.65 0.30 2.15 82.91 11.99

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 100.00 1136.46 503.50 11376 5040

avg-cpu: %user %nice %sys %iowait %idle
1.00 6.25 2.15 89.70 0.90

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 170.17 4106.51 388.39 41024 3880

avg-cpu: %user %nice %sys %iowait %idle
0.75 0.15 1.75 73.70 23.65

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 234.60 5107.20 232.80 51072 2328

avg-cpu: %user %nice %sys %iowait %idle
0.15 0.00 0.65 49.48 49.73

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 175.52 1431.37 122.28 14328 1224

avg-cpu: %user %nice %sys %iowait %idle
0.15 0.00 0.55 50.22 49.08

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 173.50 1464.00 119.20 14640 1192

avg-cpu: %user %nice %sys %iowait %idle
2.00 0.00 0.60 76.18 21.22

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 130.60 1044.80 203.20 10448 2032

avg-cpu: %user %nice %sys %iowait %idle
0.90 0.10 0.75 97.55 0.70

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 92.09 1024.22 197.80 10232 1976

avg-cpu: %user %nice %sys %iowait %idle
0.25 0.00 0.40 73.78 25.57

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 92.81 582.83 506.99 5840 5080

avg-cpu: %user %nice %sys %iowait %idle
0.20 0.00 0.55 98.85 0.40

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 90.80 657.60 383.20 6576 3832

avg-cpu: %user %nice %sys %iowait %idle
16.46 0.00 4.25 77.09 2.20

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 99.60 1174.83 549.85 11760 5504

avg-cpu: %user %nice %sys %iowait %idle
8.05 0.00 2.60 56.92 32.43

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 172.30 2063.20 128.00 20632 1280

avg-cpu: %user %nice %sys %iowait %idle
20.84 0.00 4.75 52.82 21.59

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 174.30 1416.80 484.00 14168 4840

avg-cpu: %user %nice %sys %iowait %idle
1.30 0.00 1.60 56.93 40.17

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 181.02 2858.74 418.78 28616 4192

avg-cpu: %user %nice %sys %iowait %idle
19.17 0.00 4.44 49.78 26.61

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 162.20 1286.40 373.60 12864 3736

avg-cpu: %user %nice %sys %iowait %idle
0.15 0.00 0.60 50.85 48.40

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 178.08 1436.64 97.70 14352 976

> Regards,
> Jeff Baker