A few days ago, for whatever reason I found myself repeatedly compiling linux kernels in order to solve some ext3 performance problem (my theory was that kernels after 2.6.18 have very high wait io for ext3 filesystems). This was on my linux server with lots of disk drives. Three of the drives are 300GB ones in a software RAID5 with LVM2 on top. The root filesystem is obviously not part of this. I have a small (non-mirrored) /boot and root is an LVM2 volume on top of a raw un-mirrored disk partition.

Unmirrored root? Yeah, what was I thinking.

Of course in the midst of all my reboots, the box decided not to come up. I’d had a few kernel builds that failed to boot so I assumed it was my fault, but I started to see funny DMA errors on the drive that has the root fs on it. Oh no.

And the drive was making strange noises too. This is never a good sign.

So then I set about to save the root fs. The drive was not completely dead, so  I  plugged it into a USB enclosure and had my T42 running dd_rescue against it. This eventually got all of my root partiton (minus 120Kbtyes). I fsck’d it and it seems reasonable. I really need to copy it file by file somewhere as there are bound to be files with holes in them.

Next problem. So I go and buy a new drive (a 750GB Samsung SATA), but in the course of my troubleshooting and multiple boots off CD (debian etch disk 1), my RAID5 arrays get into a bit of a weird state. Now, I had partitioned these disks into 50GB chunks and then RAIDed each of those (there is a method to this madness which I won’t go into here). So I have md5 which gives 100GB useable (50 + 50 + another 50 for parity), md6 is set up in a similar way, then md7 through to md10.

The weird state is that with the two working drives left, md9 and md10 say they are inactive when I do a ‘cat /proc/mdstat’. If I do a ‘mdadm –detail /dev/md9’ I get a state of ‘Active, Degraded, Not started’. Trying a ‘mdadm –run /dev/md9’ gives me an IO error. I also tried a ‘mdadm –assemble –force /dev/md9 /dev/sda9 /dev/sdb9 missing’ , but that just tells me the array is already active.

I check the Event id on the two partitions left in md9 (mdadm -E /dev/sda9 … mdadm -E /dev/sdb9) and they give me the same number, so in the theory the array should come up with the two drives.

So I trawl the net, and end up partitioning up my replacement drive and trying to add it in. So I tried a ‘mdadm /dev/md9 -a /dev/sdc9’. That added the drive in, but no resyncing started, and md9 stayed in its useless state.

More trawling the net, and I tried:

echo clean > /sys/block/md9/md/array_state

Finally, the array started syncing.  So now I’ve gone through and added in the all the extra partitions I’ve set up on the new drive (eg.’mdadm /dev/md5 -a /dev/sdc5′, ‘mdadm /dev/md6 -a /dev/sdc6’ … and so on).

Did a vgscan to make sure I could see my volume group. Then a ‘modprobe dm-mod’ as I was still booted off the Debian Etch install CD here (I just booted it as though I was doing an install, then went to manual partitioning, and chose the Software RAID option. Then don’t do anything further in the debian installer, hit alt-f2 and get another shell running. Now I can do cat /proc/mdstat and other stuff from this other terminal).

I had to do a vgchange volgroupname -a y

vgdisplay -v should now show lots of logical volumes. Now I go through and e2fsck -f /dev/volgroup/logicalvol for each of the logical vols. This takes quite some time, but its a safety check for me to make sure it all appears to be OK (I realise e2fsck is not going to do a complete check). It all checks out OK.