File Systems Compared

All tests are with bonnie++ 1.03a

Main components of system:
16 WD Raptor 150GB 10000 RPM drives all in a RAID 10
ARECA 1280 PCI-Express RAID adapter with 1GB BB Cache (Thanks for the
recommendation, Ron!)
32 GB RAM
Dual Intel 5160 Xeon Woodcrest 3.0 GHz processors
OS: SUSE Linux 10.1

All runs are with the write cache disabled on the hard disks, except
for one additional test for xfs where it was enabled. I tested with
ordered and writeback journaling modes for ext3 to see if writeback
journaling would help over the default of ordered. The 1GB of battery
backed cache on the RAID card was enabled for all tests as well.
Tests are in order of increasing random seek performance. In my tests
on this hardware, xfs is the decisive winner, beating all of the
other file systems in performance on every single metric. 658 random
seeks per second, 433 MB/sec sequential read, and 350 MB/sec
sequential write seems decent enough, but not as high as numbers
other people have suggested are attainable with a 16 disk RAID 10.
350 MB/sec sequential write with disk caches enabled versus 280 MB/
sec sequential write with disk caches disabled sure makes enabling
the disk write cache tempting. Anyone run their RAIDs with disk
caches enabled, or is this akin to having fsync off?

ext3 (writeback data journaling mode):
/usr/local/sbin/bonnie++ -d bonnie -s 64368:8k
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --
Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %
CP /sec %CP
hulk4 64368M 78625 91 279921 51 112346 13 89463 96 417695
22 545.7 0
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --
Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %
CP /sec %CP
16 5903 99 +++++ +++ +++++ +++ 6112 99 +++++ ++
+ 18620 100
hulk4,64368M,
78625,91,279921,51,112346,13,89463,96,417695,22,545.7,0,16,5903,99,+++
++,+++,+++++,+++,6112,99,+++++,+++,18620,100

ext3 (ordered data journaling mode):
/usr/local/sbin/bonnie++ -d bonnie -s 64368:8k
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --
Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %
CP /sec %CP
hulk4 64368M 74902 89 250274 52 123637 16 88992 96 417222
23 548.3 0
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --
Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %
CP /sec %CP
16 5941 97 +++++ +++ +++++ +++ 6270 99 +++++ ++
+ 18670 99
hulk4,64368M,
74902,89,250274,52,123637,16,88992,96,417222,23,548.3,0,16,5941,97,+++
++,+++,+++++,+++,6270,99,+++++,+++,18670,99

reiserfs:
/usr/local/sbin/bonnie++ -d bonnie -s 64368:8k
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --
Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %
CP /sec %CP
hulk4 64368M 81004 99 269191 50 128322 16 87865 96 407035
28 550.3 0
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --
Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %
CP /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ ++
+ +++++ +++
hulk4,64368M,
81004,99,269191,50,128322,16,87865,96,407035,28,550.3,0,16,+++++,+++,+
++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

jfs:
/usr/local/sbin/bonnie++ -d bonnie/ -s 64368:8k
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --
Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %
CP /sec %CP
hulk4 64368M 73246 80 268886 28 110465 9 89516 96 413897
21 639.5 0
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --
Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %
CP /sec %CP
16 3756 5 +++++ +++ +++++ +++ 23763 90 +++++ ++
+ 22371 70
hulk4,64368M,
73246,80,268886,28,110465,9,89516,96,413897,21,639.5,0,16,3756,5,++++
+,+++,+++++,+++,23763,90,+++++,+++,22371,70

xfs (with write cache disabled on disks):
/usr/local/sbin/bonnie++ -d bonnie/ -s 64368:8k
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --
Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %
CP /sec %CP
hulk4 64368M 90621 99 283916 35 105871 11 88569 97 433890
23 644.5 0
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --
Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %
CP /sec %CP
16 28435 95 +++++ +++ 28895 82 28523 91 +++++ ++
+ 24369 86
hulk4,64368M,
90621,99,283916,35,105871,11,88569,97,433890,23,644.5,0,16,28435,95,++
+++,+++,28895,82,28523,91,+++++,+++,24369,86

xfs (with write cache enabled on disks):
/usr/local/sbin/bonnie++ -d bonnie -s 64368:8k
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --
Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %
CP /sec %CP
hulk4 64368M 90861 99 348401 43 131887 14 89412 97 432964
23 658.7 0
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --
Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %
CP /sec %CP
16 28871 90 +++++ +++ 28923 91 30879 93 +++++ ++
+ 28012 94
hulk4,64368M,
90861,99,348401,43,131887,14,89412,97,432964,23,658.7,0,16,28871,90,++
+++,+++,28923,91,30879,93,+++++,+++,28012,94