2009-07-16/17.

Added new disk to raid5 linux-md array on `dint'.
Already done this before (1yr ago, on `serv') but
took no notes then.  It's pretty easy, anyway.
It can all be done on-line, but in fact the adding 
of the disk was done off-line, since other attempts
with normal (rather than slotted, hotswap) hardware
have resulted in other disks getting lost during the
jiggling of cables.

The new disk is called /dev/sdd 
Disks /dev/sd[abc] already existed.
All 4 were exactly the same length (fortunately).
All existing disks had four partitions:
	1: boot (all copies of sda1)
	2: swap (only sda2 actually used)
	3: raid (md0: root fs)
	4: raid (md1: /home)

Make the new disk have the same partition table:
# dd if=/dev/sdc of=/dev/sdd bs=512 count=1

Update kernel's idea of partitions, by writing ('w') from fdisk:
# fdisk /dev/sdd

Add new partition, and tell it to grow MD device:
# mdadm  --add /dev/md0  /dev/sdd3
# mdadm  --grow /dev/md0 -l 5 -n 4

Discover that it's going very slowly:
# cat /proc/mdstat

Set speeds (the min one has greater effect):
# echo 10000 > /proc/sys/dev/raid/speed_limit_min
# echo 40000 > /proc/sys/dev/raid/speed_limit_max

(When finished)
Make filesystem fill the new space:
# resize2fs -p /dev/md0

Likewise for the bigger one, md1.
Then it's all done.

# cat /proc/mdstat
md1 : active raid5 sdd4[3] sdc4[1] sdb4[2] sda4[0]
      1434017856 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
md0 : active raid5 sdd3[3] sdc3[1] sdb3[2] sda3[0]
      29326464 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]

# df -h | grep /md
/dev/md0               28G   11G   16G  41% /
/dev/md1              1.2T  874G  242G  79% /home

(
earlier [before resize and some copying of new stuff], had had:
/dev/md0               19G   11G  6.7G  62% /
/dev/md1              898G  762G  127G  86% /home
)

# hdparm -tT /dev/md0
 Timing cached reads:   1216 MB in  2.00 seconds = 607.73 MB/sec
 Timing buffered disk reads:  470 MB in  3.01 seconds = 156.38 MB/sec
# hdparm -tT /dev/md1
 Timing cached reads:   1216 MB in  2.00 seconds = 607.85 MB/sec
 Timing buffered disk reads:  440 MB in  3.01 seconds = 146.27 MB/sec