Adding Software RAID to a Proxmox VE 2.0 Install
Almost two years ago, I documented the process I went through to get Linux software RAID setup on Proxmox VE because Proxmox didn't officially support software RAID (which Linux does, of course). Now the fine folks at Proxmox released Proxmox VE 2.0 adding even greater features to Prox, like high-availability settings, REST api for management, etc… However, Proxmox VE 2.0 still doesn't officially support software RAID out of the box, so I figured it was time to go find out how to do it on this new version… As it turns out, due to the newer version of debian running under VE 2.0, the process is even easier than it was before! Read on and find out how.
In case you haven't read the original article, here is the common-sense disclaimer I put up for those of you out there that are more risk-prone:
WARNING: If common-sense hasn't kicked-in by now, make sure you DON'T do this on a system that is in use. Migrate your virtuals to another box or stop the virtuals and endure the downtime during the process.
Like I said above, the process is now simpler and it is reduced to 4 steps, as follows:
- Install required software
- Prepare raid devices
- Get /boot ready on /dev/md0
- Move the PVE LVM to /dev/md1
So let's get started!
1. Install required software
All you really need here is the mdadm package. I also install vim because I like it better than plain vi. On Proxmox VE 1.0 you also needed the initramfs-tools, but this is already installed on VE 2.0. So, to get your system ready, type the following on the command line of your Proxmox setup:
apt-get update; apt-get install mdadm vim
The mdadm package will prompt you for information and all you need to do on that screen is press the ENTER key.
2. Prepare raid devices
Now that you have the right software setup, let's get your raid devices ready. We will copy the partition information from /dev/sda into /dev/sdb, convert the partitions on sdb to raid members, initialize the raid devices and then save the raid configuration on /etc/mdadm/mdadm.conf so it persists after reboot. All that is done with the following 6 lines:
sfdisk -d /dev/sda | sfdisk -f /dev/sdb sfdisk -c /dev/sdb 1 fd sfdisk -c /dev/sdb 2 fd mdadm --create -l 1 -n 2 /dev/md0 missing /dev/sdb1 mdadm --create -l 1 -n 2 /dev/md1 missing /dev/sdb2 mdadm --detail --scan >> /etc/mdadm/mdadm.conf
3. Get /boot ready on /dev/md0
This is by far the step with the most number of commands, but we got quite a bit to do here… It may be possible to shorten some of these steps, but I wasn't successful on my attempts to simplify this and the instructions below worked pretty well for me. Anyway, let's begin by formatting and populating /dev/md0 with the contents of /boot.
mkfs.ext3 /dev/md0 mkdir /mnt/md0 mount /dev/md0 /mnt/md0 cp -ax /boot/* /mnt/md0
Now let's tell our system to use /dev/md0 after the Linux bootstrap (i.e.: grub2 will still use sda1 to boot for now).
vim /etc/fstab
Replace the line that has UUID=<your UUID here> /boot ext3 defaults 0 1 with /dev/md0 /boot ext3 defaults 0 1. If you are unfamiliar with vim, use the arrow keys to navigate to the line above, hit yypi# to make a copy of the old line and then turn the copy into a comment, use the up-arrow to go to the uncommented line, delete all the UUID=bla text and add /dev/md0. After that is done, hit the ESC key, which will take you out of insert mode, then :wq followed by the ENTER key. That will save and quit the file for you. We are now done with vim! Once you are on the command line again, reboot with the following command:
reboot
After the system reboots, let's verify that it is using the md0 device as your /boot mount point by typing the following:
mount|grep boot
You should get something like this:
/dev/md0 on /boot type ext3 (rw)
Now we go on to tell grub to use that device during boot as well on the next 10 commands:
echo '# customizations' >> /etc/default/grub echo 'GRUB_DISABLE_LINUX_UUID=true' >> /etc/default/grub echo 'GRUB_PRELOAD_MODULES="raid dmraid"' >> /etc/default/grub echo raid1 >> /etc/modules echo raid1 >> /etc/initramfs-tools/modules grub-install /dev/sda grub-install /dev/sdb grub-install /dev/md0 update-grub update-initramfs -u
Maybe you don't need all 3 grub-install commands, but for me, not having the last one there didn't work and when reverting the process during one of my tests, I ended-up having to reissue the command grub-install /dev/sda.
We are almost done with this step! All that remains to do is make /dev/sda1 part of /dev/md0 and reboot using this config to make sure all is working at it should. We get that done with the following 2 commands:
sfdisk -c /dev/sda 1 fd mdadm --add /dev/md0 /dev/sda1
This will get /dev/md0 rebuilding, and it shouldn't take long. You can verify the progress of the process with the following command:
watch -n 5 cat /proc/mdstat
Once that is completed, reboot and we are done with this step!
reboot
4. Move the PVE LVM to /dev/md1
Now the process is pretty much similar to the old one. This step is simple, but it is the one that could take the longest time to complete, depending on how big your data partition is and how fast is your system. What we need to do is vacate /dev/sda2 so we can join it to /dev/md1, and we do this with the following commands: (warning: the pvmove command can take a long time to complete, so use it on a tty or inside a screen session)
pvcreate /dev/md1 vgextend pve /dev/md1 pvmove /dev/sda2 /dev/md1 vgreduce pve /dev/sda2 pvremove /dev/sda2 sfdisk --change-id /dev/sda 2 fd mdadm --add /dev/md1 /dev/sda2
You will get your second raid rebuilding after the last mdadm command. This will take longer than the 1st one and you can check its progress the same way as before, with:
watch -n 5 cat /proc/mdstat
However, this time it may be good to boost the limits with which the RAID subsystem can read and write to its devices. You do that with the following commands:
echo 800000 > /proc/sys/dev/raid/speed_limit_min echo 1600000 > /proc/sys/dev/raid/speed_limit_max
Hopefully this guide will save you some time and quite possibly a lot of head-ache and frustration! If you like it or if you have a suggestion to improve on it in any way, please leave me a comment below.
Easy Peasy!
-PCP
- peter's blog
- Login or register to post comments
