Re: Using pgiosim realistically

On Fri, May 13, 2011 at 09:09:41PM +0000, John Rouillard wrote:
> Hi all:
>
> I am adding pgiosim to our testing for new database hardware and I am
> seeing something I don't quite get and I think it's because I am using
> pgiosim incorrectly.
>
> Specs:
>
> OS: centos 5.5 kernel: 2.6.18-194.32.1.el5
> memory: 96GB
> cpu: 2x Intel(R) Xeon(R) X5690 @ 3.47GHz (6 core, ht enabled)
> disks: WD2003FYYS RE4
> raid: lsi - 9260-4i with 8 disks in raid 10 configuration
> 1MB stripe size
> raid cache enabled w/ bbu
> disk caches disabled
> filesystem: ext3 created with -E stride=256
>
> I am seeing really poor (70) iops with pgiosim. According to:
> http://www.tomshardware.com/reviews/2tb-hdd-7200,2430-8.html in the
> database benchmark they are seeing ~170 iops on a single disk for
> these drives. I would expect an 8 disk raid 10 should get better then
> 3x the single disk rate (assuming the data is randomly distributed).
>
> To test I am using 5 100GB files with
>
> sudo ~/pgiosim -c -b 100G -v file?
>
> I am using 100G sizes to make sure that the data read and files sizes
> exceed the memory size of the system.
>
> However if I use 5 1GB files (and still 100GB read data) I see 200+ to
> 400+ iops at 50% of the 100GB of data read, which I assume means that
> the data is cached in the OS cache and I am not really getting hard
> drive/raid I/O measurement of iops.
>
> However, IIUC postgres will never have an index file greater than 1GB
> in size
> (http://www.postgresql.org/docs/8.4/static/storage-file-layout.html)
> and will just add 1GB segments, so the 1GB size files seems to be more
> realistic.
>
> So do I want 100 (or probably 2 or 3 times more say 300) 1GB files to
> feed pgiosim? That way I will have enough data that not all of it can
> be cached in memory and the file sizes (and file operations:
> open/close) more closely match what postgres is doing with index
> files?
>
> Also in the output of pgiosim I see:
>
> 25.17%, 2881 read, 0 written, 2304.56kB/sec 288.07 iops
>
> which I interpret (left to right) as the % of the 100GB that has been
> read, the number of read operations over some time period, number of
> bytes read/written and the io operations/sec. Iops always seems to be
> 1/10th of the read number (rounded up to an integer). Is this
> expected and if so anybody know why?
>
> While this is running if I also run "iostat -p /dev/sdc 5" I see:
>
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sdc 166.40 2652.80 4.80 13264 24
> sdc1 2818.80 1.20 999.20 6 4996
>
> which I am interpreting as 2818 read/io operations (corresponding more
> or less to read in the pgiosim output) to the partition and of those
> only 116 are actually going to the drive??? with the rest handled from
> OS cache.
>
> However the tps isn't increasing when I see pgiosim reporting:
>
> 48.47%, 4610 read, 0 written, 3687.62kB/sec 460.95 iops
>
> an iostat 5 output near the same time is reporting:
>
> Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
> sdc 165.87 2647.50 4.79 13264 24
> sdc1 2812.97 0.60 995.41 3 4987
>
> so I am not sure if there is a correlation between the read and tps
> settings.
>
> Also I am assuming blks written is filesystem metadata although that
> seems like a lot of data
>
> If I stop the pgiosim, the iostat drops to 0 write and reads as
> expected.
>
> So does anybody have any comments on how to test with pgiosim and how
> to correlate the iostat and pgiosim outputs?
>
> Thanks for your feedback.
> --
> -- rouilj
>
> John Rouillard System Administrator
> Renesys Corporation 603-244-9084 (cell) 603-643-9300 x 111
>

Hi John,

Those drives are 7200 rpm drives which would give you a maximum write
rate of 120/sec at best with the cache disabled. I actually think your
70/sec is closer to reality and what you should anticipate in real use.
I do not see how they could make 170/sec. Did they strap a jet engine to
the drive. :)

Regards,
Ken