2006-09-30 -- serv.ntmm.org New disks! Was a single 160GB PATA for system and general programs, then two old 40GB PATAs in linux-md raid1 for /home, and then a single 280GB SATA for /home/public (multimedia etc.). Now have swapped the big sata and the two raid5 patas for three similar 320GB Hitachi satas, in raid5. Owing partly to that strange feature of raid1 that no implementation yet met increases the speed over a single disk, this new system was much faster: more than double the read-speed. This is desirable, as the Gb network can take this, and some uses such as audio/video processing really derive an advantage # cat /proc/mdstat Personalities : [raid5] [raid4] md0 : active raid5 sdc[2] sdb[1] sda[0] 625142272 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU] unused devices: -------------------------------------------------------------------------- Testing on the server -- serv.ntmm.org -------------------------------------------------------------------------- # hdparm -tT /dev/sda: Timing cached reads: 2384 MB in 2.00 seconds = 1189.77 MB/sec Timing buffered disk reads: 206 MB in 3.01 seconds = 68.52 MB/sec /dev/sdb: Timing cached reads: 2228 MB in 2.00 seconds = 1112.69 MB/sec Timing buffered disk reads: 206 MB in 3.00 seconds = 68.59 MB/sec /dev/sdc: Timing cached reads: 2372 MB in 2.00 seconds = 1185.89 MB/sec Timing buffered disk reads: 214 MB in 3.01 seconds = 71.20 MB/sec /dev/md0: Timing cached reads: 2232 MB in 2.00 seconds = 1114.97 MB/sec Timing buffered disk reads: 410 MB in 3.00 seconds = 136.52 MB/sec (this hdparm got very close (~3% results) when re-run) # bonnie++ -n 16 -s 4g -u nt:users -d /srv/home/public/tmp (this directory is on the /dev/md0 filesystem) Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP serv 4G 267 93 54098 21 22294 8 903 98 94496 21 228.2 6 Latency 125ms 1561ms 659ms 38358us 199ms 451ms Version 1.93c ------Sequential Create------ --------Random Create-------- serv -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 1916 94 +++++ +++ +++++ +++ 2135 96 +++++ +++ 5417 93 Latency 18570us 162us 93us 85558us 162us 707us So, about 95MBps reading blocks, 2000 files/sec created, 5000 files/sec deleted -------------------------------------------------------------------------- Testing on a client (Gb ethernet, one switch inbetween, TCP,NFS3,async) -------------------------------------------------------------------------- # bonnie++ -n 16 -s 4g -u nt:users -d /home/public/tmp (this directory in mounted from an NFs export of the serv:/dev/md0 filesystem) Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP box 4G 4 12 13875 12 10004 14 744 90 52859 14 235.6 9 Latency 2766ms 451ms 240ms 72898us 83583us 500m Version 1.93c ------Sequential Create------ --------Random Create-------- box -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 586 6 2874 14 1274 10 632 7 3071 14 982 7 Latency 442ms 15943us 69633us 58741us 10982us 106ms Still a large large difference _even_ when doing continuous things such as long reads. More pronounced difference for many creations/deletions (many remote calls). -------------------------------------------------------------------------- Testing on same client (Gb ethernet, one switch inbetween, NFSv4,async) -------------------------------------------------------------------------- bonnie++ -n 16 -s 4g -u nt:users -d /home/public/scratch (new name for same thing) Version 1.93c ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP box 4G 4 12 15112 7 10230 5 996 92 67407 14 236.9 5 Latency 2504ms 307ms 303ms 52052us 120ms 424ms Version 1.93c ------Sequential Create------ --------Random Create-------- box -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 574 3 2532 8 1351 7 598 4 2545 8 1072 5 Latency 140ms 10338us 96029us 114ms 10377us 88545us