Re: Slow queries / commits, mis-configuration or hardware issues?

On Tue, Nov 15, 2011 at 5:16 PM, Tomas Vondra <tv(at)fuzzy(dot)cz> wrote:
> Dne 14.11.2011 22:58, Cody Caughlan napsal(a):
>> I ran bonnie++ on a slave node, doing active streaming replication but
>> otherwise idle:
>> http://batch-files-test.s3.amazonaws.com/sql03.prod.html
>>
>> bonnie++ on the master node:
>> http://batch-files-test.s3.amazonaws.com/sql01.prod.html
>>
>> If I am reading this right, this is my first time using it, the
>> numbers dont look too good.
>
> I've done some benchmarks on my own (m1.xlarge instance), and the
> results are these (http://pastebin.com/T1LXHru0):
>
> single drive
> ------------
> dd writes: 62 MB/s
> dd reads: 110 MB/s
> bonnie seq. writes: 55 MB/s
> bonnie seq. rewrite: 33 MB/s
> bonnie seq. reads: 91 MB/s
> bonnie seeks: 370/s
>
> raid 0 (4 devices)
> -----------------------------
> dd writes: 220 MB/s
> dd reads: 380 MB/s
> bonnie seq. writes: 130 MB/s
> bonnie seq. rewrite: 114 MB/s
> bonnie seq. reads: 280 MB/s
> bonnie seeks: 570/s
>
> raid 10 (4 devices)
> -----------------------------
> dd writes: 90 MB/s
> dd reads: 200 MB/s
> bonnie seq. writes: 49 MB/s
> bonnie seq. rewrite: 56 MB/s
> bonnie seq. reads: 160 MB/s
> bonnie seeks: 590/s
>

Interesting. I spun up a new m1.xlarge and did the same RAID10 config
(4 drives) except with a chunk size of 512K (instead of 256K) and the
machine was completely idle. Bonnie:

http://batch-files-test.s3.amazonaws.com/idle-512k-chunk.html

Which has similar-ish performance as yours, except for worse seeks but
a bit better seq. reads.

The other bonnies I sent over were NOT on idle systems. This one is
the master, which receives a heavy stream of writes and some reads

http://batch-files-test.s3.amazonaws.com/sql01.prod.html

And this is the slave, which is all writes and no reads:
http://batch-files-test.s3.amazonaws.com/sql03.prod.html

How did you build your RAID array? Maybe I have a fundamental flaw /
misconfiguration. I am doing it via:

$ yes | mdadm --create /dev/md0 --level=10 -c256 --raid-devices=4
/dev/xvdb /dev/xvdc /dev/xvdd /dev/xvde
$ pvcreate /dev/md0
$ vgcreate lvm-raid10 /dev/md0
$ lvcreate -l 215021 lvm-raid10 -n lvm0
$ blockdev --setra 65536 /dev/lvm-raid10/lvm0
$ mkfs.xfs -f /dev/lvm-raid10/lvm0
$ mkdir -p /data && mount -t xfs -o noatime /dev/lvm-raid10/lvm0 /data