2010-05-08. == Purpose: Something like this: dd bs=4096 /dev/sdc had been used for copying the contents of a 1TB disk to a replacement disk, but the speed soon ended up at only about 47MBps, which is considerably (tens of percent) less than was expected for the sustainable read or write speed. == Stuff: /dev/sda Device Model: Hitachi HDT721010SLA360 Serial Number: STF604MR2MKGWP Firmware Version: ST6OA31B User Capacity: 1,000,204,886,016 bytes /dev/sdc Device Model: Hitachi HDS721010CLA332 Serial Number: JP2921HQ24UB8A Firmware Version: JP4OA39C User Capacity: 1,000,204,886,016 bytes sda is on an onboard sata controller, along with sdb: driver: sata_via 00:0f.0 IDE interface: VIA Technologies, Inc. VIA VT6420 SATA RAID Controller (rev 80) sdc is on a PCI-card sata controller, along with sdd: driver: sata_promise 00:0c.0 Mass storage controller: Promise Technology, Inc. PDC40775 (SATA 300 TX2plus) (rev 02) The motherboard is VIA-based, from mid-2005. The CPU is AMD Athlon(tm) 64 Processor 3700+, 2.2GHz. RAM is 1GB. Kernel is Linux 2.6.18-164.6.1.el5 #1 SMP Tue Oct 27 11:28:30 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux , on RHEL5. dd version is GNU coreutils 5.97. == Results: --------- for bs in 512 4096 16384 65536 262144 1048576 4194304 8225280 do echo " "; echo "blocksize: $bs"; dd bs=$bs /dev/sdc & for n in `seq 1 8` do echo -n "$n " killall -USR1 dd sleep 2 done killall -TERM dd done blocksize rate (MB/s) 512 20 4096 106 16384 106 65536 105 262144 83 1048576 77 4194304 75 8225280 50 ---------- The same was done again, with the in/out file-redirects replaced with if= and of= arguments, dd bs=$bs if=/dev/sda of=/dev/sdc. The result was indistinguishable in terms of speed, but (perhaps by chance, perhaps through greater time needed for interuption due to doing its own buffering?) the cases at 4096 and 262144 blocksize didn't happen, as their own dd had apparently started before the TERM signal to all dd had happened (a bit strange). Try a delay after that, next time. ---------- The next time used a pipeline, following `on web' advice about making dd quicker: the claim was that a single dd process will read, then write, and so on, rather than keeping both disks working hard. This might well be a problem only with old proprietary dds. A thorough kill was used, including a wait. for bs in 512 4096 16384 65536 262144 1048576 4194304 8225280 ; do echo " " ; echo "blocksize: $bs"; dd bs=$bs if=/dev/sda | dd bs=$bs of=/dev/sdc & for n in `seq 1 8` ; do echo -n "$n "; killall -USR1 dd ; sleep 2 ; done ; killall -TERM dd ; sleep 8 ; killall -KILL dd ; sleep 8 ; done At bs=262144 the speed stayed around 105 rather than falling to 80. At bs=8225280 the speed stayed around 70 rather than falling to 50. At bs=512 the speed was even a little worse, about 17 rather than 20. At the `best' points, e.g. 4096 and rather bigger, the speed in all cases was about 106, which is also known from hdparm to be about the sustained read-speed of the disk. ------------ It seems then that the pipelining of dd and use of redirects versus file arguments makes no difference when a good block-size is used, and that 4096 is a good blocksize. The website had said that 512bytes is the default. Is it? Repeating the last case (pipelined) with no bs= argument, the speeds were seen to be very similar to those seen before for 512B. Using a blocksize of 1B gave about 0.4MBps. So: setting the block-size several times bigger than the default (perhaps to the actual blocksize of a modern disk?) increased speed by over 5 times, but further increase can be bad. Testing a transfer of well over the computer's memory would be advisable next time, for thoroughness. For today, we've `seen enough'. --------------------- Raw data (console log). [root@backup ~]# for bs in 512 4096 16384 65536 262144 1048576 4194304 8225280 ; do echo " " ; echo "blocksize: $bs"; dd bs=$bs /dev/sdc & for n in `seq 1 8` ; do echo -n "$n "; killall -USR1 dd ; sleep 2 ; done ; killall -TERM dd ; done blocksize: 512 [1] 23225 1 3201+0 records in 3201+0 records out 1638912 bytes (1.6 MB) copied, 0.086028 seconds, 19.1 MB/s 2 88105+0 records in 88105+0 records out 45109760 bytes (45 MB) copied, 2.09666 seconds, 21.5 MB/s 3 173641+0 records in 173641+0 records out 88904192 bytes (89 MB) copied, 4.1175 seconds, 21.6 MB/s 4 247641+0 records in 247641+0 records out 126792192 bytes (127 MB) copied, 6.14044 seconds, 20.6 MB/s 5 314425+0 records in 314425+0 records out 160985600 bytes (161 MB) copied, 8.16321 seconds, 19.7 MB/s 6 379273+0 records in 379273+0 records out 194187776 bytes (194 MB) copied, 10.1848 seconds, 19.1 MB/s 7 445385+0 records in 445385+0 records out 228037120 bytes (228 MB) copied, 12.2066 seconds, 18.7 MB/s 8 512329+0 records in 512329+0 records out 262312448 bytes (262 MB) copied, 14.2365 seconds, 18.4 MB/s blocksize: 4096 [2] 23244 1 [1]- Terminated dd bs=$bs < /dev/sda > /dev/sdc 2 27517+0 records in 27516+0 records out 112705536 bytes (113 MB) copied, 1.06579 seconds, 106 MB/s 3 78270+0 records in 78269+0 records out 320589824 bytes (321 MB) copied, 3.07788 seconds, 104 MB/s 4 131646+0 records in 131645+0 records out 539217920 bytes (539 MB) copied, 5.08986 seconds, 106 MB/s 5 184984+0 records in 184983+0 records out 757690368 bytes (758 MB) copied, 7.09889 seconds, 107 MB/s 6 238238+0 records in 238237+0 records out 975818752 bytes (976 MB) copied, 9.10759 seconds, 107 MB/s 7 291478+0 records in 291477+0 records out 1193889792 bytes (1.2 GB) copied, 11.1174 seconds, 107 MB/s 8 343805+0 records in 343804+0 records out 1408221184 bytes (1.4 GB) copied, 13.1276 seconds, 107 MB/s blocksize: 16384 [3] 23263 1 [2]- Terminated dd bs=$bs < /dev/sda > /dev/sdc 2 6735+0 records in 6734+0 records out 110329856 bytes (110 MB) copied, 1.03129 seconds, 107 MB/s 3 19310+0 records in 19309+0 records out 316358656 bytes (316 MB) copied, 3.04599 seconds, 104 MB/s 4 32559+0 records in 32558+0 records out 533430272 bytes (533 MB) copied, 5.05687 seconds, 105 MB/s 5 45763+0 records in 45762+0 records out 749764608 bytes (750 MB) copied, 7.06885 seconds, 106 MB/s 6 58999+0 records in 58998+0 records out 966623232 bytes (967 MB) copied, 9.0776 seconds, 106 MB/s 7 72199+0 records in 72199+0 records out 1182908416 bytes (1.2 GB) copied, 11.0866 seconds, 107 MB/s 8 85271+0 records in 85270+0 records out 1397063680 bytes (1.4 GB) copied, 13.0983 seconds, 107 MB/s blocksize: 65536 [4] 23282 1 [3]- Terminated dd bs=$bs < /dev/sda > /dev/sdc 2 1545+0 records in 1544+0 records out 101187584 bytes (101 MB) copied, 0.961984 seconds, 105 MB/s 3 4679+0 records in 4678+0 records out 306577408 bytes (307 MB) copied, 2.97186 seconds, 103 MB/s 4 7957+0 records in 7956+0 records out 521404416 bytes (521 MB) copied, 4.98271 seconds, 105 MB/s 5 11229+0 records in 11228+0 records out 735838208 bytes (736 MB) copied, 6.99381 seconds, 105 MB/s 6 14547+0 records in 14546+0 records out 953286656 bytes (953 MB) copied, 9.00431 seconds, 106 MB/s 7 17832+0 records in 17832+0 records out 1168637952 bytes (1.2 GB) copied, 11.0138 seconds, 106 MB/s 8 21075+0 records in 21074+0 records out 1381105664 bytes (1.4 GB) copied, 13.0218 seconds, 106 MB/s blocksize: 262144 [5] 23301 1 [4]- Terminated dd bs=$bs < /dev/sda > /dev/sdc 2 289+0 records in 288+0 records out 75497472 bytes (75 MB) copied, 0.877744 seconds, 86.0 MB/s 3 921+0 records in 920+0 records out 241172480 bytes (241 MB) copied, 2.89261 seconds, 83.4 MB/s 4 1550+0 records in 1549+0 records out 406061056 bytes (406 MB) copied, 4.90754 seconds, 82.7 MB/s 5 2173+0 records in 2172+0 records out 569376768 bytes (569 MB) copied, 6.92611 seconds, 82.2 MB/s 6 2777+0 records in 2776+0 records out 727711744 bytes (728 MB) copied, 8.94166 seconds, 81.4 MB/s 7 3383+0 records in 3382+0 records out 886571008 bytes (887 MB) copied, 10.9558 seconds, 80.9 MB/s 8 3993+0 records in 3992+0 records out 1046478848 bytes (1.0 GB) copied, 12.9725 seconds, 80.7 MB/s blocksize: 1048576 [6] 23320 1 [5]- Terminated dd bs=$bs < /dev/sda > /dev/sdc 2 78+0 records in 77+0 records out 80740352 bytes (81 MB) copied, 0.957856 seconds, 84.3 MB/s 3 228+0 records in 227+0 records out 238026752 bytes (238 MB) copied, 2.96973 seconds, 80.2 MB/s 4 375+0 records in 374+0 records out 392167424 bytes (392 MB) copied, 4.97951 seconds, 78.8 MB/s 5 519+0 records in 518+0 records out 543162368 bytes (543 MB) copied, 6.99007 seconds, 77.7 MB/s 6 662+0 records in 661+0 records out 693108736 bytes (693 MB) copied, 9.01514 seconds, 76.9 MB/s 7 803+0 records in 802+0 records out 840957952 bytes (841 MB) copied, 11.0269 seconds, 76.3 MB/s 8 945+0 records in 944+0 records out 989855744 bytes (990 MB) copied, 13.0377 seconds, 75.9 MB/s blocksize: 4194304 [7] 23339 1 [6]- Terminated dd bs=$bs < /dev/sda > /dev/sdc 2 19+0 records in 18+0 records out 75497472 bytes (75 MB) copied, 0.997011 seconds, 75.7 MB/s 3 56+0 records in 55+0 records out 230686720 bytes (231 MB) copied, 3.03851 seconds, 75.9 MB/s 4 91+0 records in 91+0 records out 381681664 bytes (382 MB) copied, 5.02915 seconds, 75.9 MB/s 5 127+0 records in 126+0 records out 528482304 bytes (528 MB) copied, 7.06004 seconds, 74.9 MB/s 6 162+0 records in 161+0 records out 675282944 bytes (675 MB) copied, 9.08178 seconds, 74.4 MB/s 7 197+0 records in 196+0 records out 822083584 bytes (822 MB) copied, 11.0786 seconds, 74.2 MB/s 8 232+0 records in 231+0 records out 968884224 bytes (969 MB) copied, 13.0892 seconds, 74.0 MB/s blocksize: 8225280 [8] 23358 1 [7]- Terminated dd bs=$bs < /dev/sda > /dev/sdc 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.084795 seconds, 0.0 kB/s 2 13+0 records in 12+0 records out 98703360 bytes (99 MB) copied, 2.03874 seconds, 48.4 MB/s 3 25+0 records in 25+0 records out 205632000 bytes (206 MB) copied, 4.08337 seconds, 50.4 MB/s 4 38+0 records in 37+0 records out 304335360 bytes (304 MB) copied, 6.09718 seconds, 49.9 MB/s 5 50+0 records in 50+0 records out 411264000 bytes (411 MB) copied, 8.14406 seconds, 50.5 MB/s 6 62+0 records in 62+0 records out 509967360 bytes (510 MB) copied, 10.1066 seconds, 50.5 MB/s 7 75+0 records in 75+0 records out 616896000 bytes (617 MB) copied, 12.1695 seconds, 50.7 MB/s 8 88+0 records in 87+0 records out 715599360 bytes (716 MB) copied, 14.1993 seconds, 50.4 MB/s