2012-02-20. Interesting to note that a set of 5 disks (1TB, WD Caviar Green) that have been a raid5 array for a year or two in another computer (serv) give a much higher array read-speed when in a newer computer with newer SATA-controller. In both cases, the kernel is a recent linux, 3.[12].x . %% on the newer machine: % 5 disks: raw read-speed by hdparm about 100MB/s /dev/sdb: Timing buffered disk reads: 304 MB in 3.01 seconds = 101.14 MB/sec /dev/sdd: Timing buffered disk reads: 320 MB in 3.00 seconds = 106.58 MB/sec % array (raid5 on 5) gives 4 times this: about double the rate on % the oldhardware /dev/md127: Timing buffered disk reads: 1222 MB in 3.00 seconds = 407.17 MB/sec %% on the older machine: % no there are just two (bigger) disks, "raid5" (inc. 1 "missing") /dev/sda: Timing buffered disk reads: 286 MB in 3.00 seconds = 95.23 MB/sec /dev/sdb: Timing buffered disk reads: 280 MB in 3.00 seconds = 93.33 MB/sec % the raw raid device /dev/md/serv:public-b: Timing buffered disk reads: 300 MB in 3.00 seconds = 99.87 MB/sec /dev/md/serv:public-b: Timing buffered disk reads: 332 MB in 3.10 seconds = 107.06 MB/sec % the crypted [luks] version used for filesystem /dev/mapper/public: Timing buffered disk reads: 206 MB in 3.13 seconds = 65.78 MB/sec /dev/mapper/public: Timing buffered disk reads: 206 MB in 3.12 seconds = 66.01 MB/sec Yes, of course it's not a kind test, with one array having 5 members and the other two (with no redundancy). But it's interesting to see how the encryption prevents a good speed being obtained, limiting the rate to below a single disk. It's also plausible that this isn't terribly limiting for some other types of operation such as random read/write, where the seeking of the disks could be a bigger limitation than the encryption. So, a little try with bonnie++, taking "-s " as twice the RAM. The filesystems are both ext4. The 5disk array has chunks of 64K, the 3[2] disk of 512K (both were defaults at time of array creation). % the older system (with newer, 3[2] disk "ciphertext" array) root@serv # bonnie++ -d n -c 4 -s 4000 -u nt -g nt ... Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 4 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP serv 4000M 483 97 21686 4 25716 6 1779 93 123478 14 260.0 6 Latency 47496us 21806ms 11841ms 25444us 180ms 5140ms Version 1.96 ------Sequential Create------ --------Random Create-------- serv -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 10221 21 +++++ +++ 20984 36 13071 25 +++++ +++ 22540 39 Latency 1785us 1811us 3515us 2046us 39us 3569us 1.96,1.96,serv,4,1329751008,4000M,,483,97,21686,4,25716,6,1779,93,123478,14,260.0,6,16,,,,,10221,21,+++++,+++,20984,36,13071,25,+++++,+++,22540,39,47496us,21806ms,11841ms,25444us,180ms,5140ms,1785us,1811us,3515us,2046us,39us,3569us % the newer system (with older, 5disk "cleartext" array) root@simlin1 # bonnie++ -d n -c 4 -s 16000 -u nt -g users ... Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 4 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP simlin1 16000M 1152 96 44161 4 38074 2 4841 83 150653 5 499.4 5 Latency 6860us 96180ms 1269ms 31195us 171ms 126ms Version 1.96 ------Sequential Create------ --------Random Create-------- simlin1 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 1450 1 +++++ +++ 14083 10 +++++ +++ +++++ +++ +++++ +++ Latency 20333us 258us 396us 66us 15us 129us 1.96,1.96,simlin1,4,1329749290,16000M,,1152,96,44161,4,38074,2,4841,83,150653,5,499.4,5,16,,,,,1450,1,+++++,+++,14083,10,+++++,+++,+++++,+++,+++++,+++,6860us,96180ms,1269ms,31195us,171ms,126ms,20333us,258us,396us,66us,15us,129us Conclusion (weak, with so many variables not controlled, e.g. chunk, age of array, hardware): the choice of crypted or not does appear to have at least much less effect on many bonnie++ parameters than is seen between simple hdparm tests; factors more like 1.5 or 2 rather than 4 are seen for superiority on the system that is clearly the winner in number of disks and newness of controller, though unencumbered by encryption processing.