Reshaping mdadm RAID 6 to RAID 10

By Paulus, 2 September, 2018

When setting up a RAID array I have always opted for either RAID 5 for arrays with less than five drives or RAID 6 when there are more than five. It wasn't until recently that I noticed how much of a penalty there is for running a RAID 6, six IOPs while the penalty for RAID 5 is four IOPs. When a drive has failed in one of my RAID 6 arrays, I never saw the rebuild speed get above 75MiB/s even though the failed drive was capable of 150MiB/s. For smaller arrays, that may not be such a big deal. When it's a 16TiB array it will take about two and a half days to rebuild. During that time, there is a lot of reading that puts stress on the other drives and an increased chance of another drive failing. That is partly why I wanted to switch to RAID 10, for the rebuild speed plus the write performance. To be clear, I understand that RAID is NOT a backup, but I'd rather not have to restore my data from a backup. The following shows how I converted my RAID 6 array to a RAID 10 array.

Before beginning, please be aware that the entire process involves a lot of waiting. During any of the lengthy RAID steps, you can view the progress by looking at the /proc/mdstat file.

Preperation

I have eight 3TiB drives, /dev/sd[a-h] with the RAID device of /dev/md1 mounted at /home. I am not using LVM with this device.

# sysctl -w dev.raid.speed_limit_max=400000

400,000K/s is fast enough for my setup since I never saw the rebuild get above 350,000K/s.

# umount /home
# e2fsck -f /dev/md1
# resize2fs -p -M /dev/md1
# e2fsck -f /dev/md1
# mdadm --grow /dev/md1 --array-size 12000G

In the end I will have 12TiB of usable disk space and must make sure that the sum of the data is less than that. Additionally, the array size must be changed with --array-size. This only causes the array to appear a different size to programs and is not a persistent change but it is required to in the final reshaping step.

Reshaping

You cannot go from RAID 6 to RAID 10 directly, so you have to work your way down to RAID 5 and then to RAID 0 before RAID 10. If you are not using RAID 6, then you can skip this.

# mdadm --grow /dev/md1 --level=5 --raid-devices=8 --backup-file=md1.bk --force
# mdadm --grow /dev/md1 --level=5 --raid-devices=8 --backup-file=md1.bk --continue
Wait for reshaping to complete
# mdadm --grow /dev/md1 --level=5 --raid-devices=8

The first command backups the data to file and starts the reshaping process. mdadm will state that you need a spare drive to avoid a degraded array. Using the --force will allow you to continue without a spare. The only difference between the first command and the second command, aside from --force is the --continue. The --continue option in conjunction with everything else continues the operation because it was interrupted during reshaping. The final command tells mdadm to update itself. The use of --continue is not necessary, but it was for me when I was going through the process. Adding it doesn't hurt anything if it's not needed.

# cat /proc/mdstat
or
# mdadm -D /dev/md1
To watch the progress
# watch -n 1 cat /proc/mdstat

Repeat the commands above but substitute 0 for 5.

# mdadm --grow /dev/md1 --level=0 --raid-devices=8 --backup-file=md1.bk
# mdadm --grow /dev/md1 --level=0 --raid-devices=8 --backup-file=md1.bk --continue
Wait for reshaping to complete
# mdadm --grow /dev/md1 --level=0 --raid-devices=8

# mdadm --grow /dev/md1 --level=10 --raid-devices=8 --backup-file=md1.bk
Wait for reshaping and recovery to complete

Finishing up

When the array has finished reshaping and rebuilding the array size and file system size need to be restored.

# e2fsck -f /dev/md1
# mdadm --grow /dev/md1 --array-size max
# resize2fs -p /dev/md1
# e2fsck -f /dev/md1

Within an Volume Group

When the array is a member of a volume group it can get a bit tricky. The first thing to do is either move all logical volumes off the physical volume or shift or resize the logical volumes so they don't span beyond the target physical volume. Unfortunately, pvresize does not do this for you at this time so you may need to resize logical volumes and move the physical extents. In the event that one or more volumes containing the OS, reshaping needs to be done from a bootable flash drive.

Tip: If you don't feel comfortable using the command line to manage logical volumes, you can use kvpm

# pvs
  PV         VG       Fmt  Attr PSize PFree
  /dev/md0   vg00     lvm2 a--  14.19t    0 
  /dev/md1   vg00     lvm2 a--  7.28t    0
# lvs -o +devices
  LV     VG       Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices        
  backup vg00     -wi-ao---- 18.36t                                                     /dev/md0(25748)
  backup vg00     -wi-ao---- 18.36t                                                     /dev/md1(0)    
  root   vg00     -wi-ao---- 93.13g                                                     /dev/md0(0)    
  swap   vg00     -wi-ao----  7.45g                                                     /dev/md0(23841)
# umount /dev/mapper/vg00-backup
# lvresize --resize-fs -L -7T vg00/backup
# pvmove /dev/md0 /dev/md1
# mdadm --grow /dev/md0 --array-size 12000G
# mdadm --grow /dev/md0 --level=5 --raid-devices=8 --backup-file=md0.bk --force
# mdadm --grow /dev/md0 --level=5 --raid-devices=8 --backup-file=md0.bk --continue
# mdadm --grow /dev/md0 --level=5 --raid-devices=8
# mdadm --grow /dev/md0 --level=0 --raid-devices=8 --backup-file=md0.bk
# mdadm --grow /dev/md0 --level=0 --raid-devices=8 --backup-file=md0.bk --continue
# mdadm --grow /dev/md0 --level=0 --raid-devices=8
# mdadm --grow /dev/md0 --level=10 --raid-devices=8
# pvresize /dev/md0
# vgextend vg00 /dev/md0
# lvresize --resize-fs -l 100%FREE vg00/backup

Lines 1 and 6 are checking to see if any of the logical volumes span more than one physical device. On lines 14 and 15 the logical volume that spans two physical volumes is resized to fit on a single physical volume. After resizing we make sure that the volume is on a single physical volume. The command on line 24 resizes the physical volume to take up the maximum size of the array. Finally on lines 25 and 26 the physical volume is added back into the group and the logical volume is resized to take up all available physical extents.

Encrypted Volumes

Resizing encrypted volumes isn't as daunting as one might think. Essentially, you decrypt the device, resize the file system within, close the encrypted volume, and reshape as normal. Once done, unlock the volume, resize the filesystem, and continue using it as you normally would.