Setting up a software RAID array: Difference between revisions

Latest revision as of 05:39, 26 August 2023

There are various forms of RAID: via a hardware RAID controller, "fake RAID", and "software RAID" using mdadm, which is Linux-only. These instructions discuss only the last form of RAID. Only RAID for arbitrary storage is covered here. It's possible to have one's system root /, or /var, or swap, or even one's /boot, on a RAID array. See Setting up disks manually for more details about doing any of that.

RAID Levels

There are several "levels" of RAID to choose between:

RAID0 essentially just glues two devices together, making a larger virtual drive. Reads and writes are "striped" between the drives for speed improvements. (That is, your hardware may read from, or write different data to, multiple devices in parallel.) A "device" here is usually a partition of a hard drive.
RAID1 "mirrors" writes to two devices, for improved safety. Then if one of the devices fails, the data will still be available on the other.
RAID5 is similar to RAID1, but it uses three devices and provides the space of two of them. The data will be preserved as long as any two of the three devices continue to work.

There are other RAID levels as well. Here is an explanation of their differences.

Advice

Your /boot partition should either not be on RAID, or else be on a RAID1 array, with no further layers of encryption or LVM. (Alpine's default bootloader extlinux can't handle either. Grub2 can handle /boot being on LVM.) The usual practice is to create a small (32--100 MB) partition for /boot. That can be a mirrored (RAID1) volume, however this is just for post-init access. That way, when you write a new kernel or bootloader config file to /boot, it gets written to multiple physical partitions. During the pre-init, bootloader phase, only one of those partitions will be used (and it will be mounted read-only).
It's important to note that extlinux can't boot from partitions created with mdadm metadata version 1.2, which is the default. To be on the safe side, it's recommended to create the partition based on an older metadata version, with the --metadata=0.90 parameter for the /boot partition.
You can put swap on a RAID0 volume, but there doesn't seem to be any good reason to do so. The Linux kernel already knows how to stripe several swap partitions. So you can just devote multiple ordinary (non-RAID) partitions to swap, and get the same effect. The downside from doing either of these things is that when one of your disks fails, the system will go down. For better reliability, create a mirrored (RAID1) volume and put swap there. This will let your system keep running if one of the disks fails.
All partitions in a RAID array should be the same size.
Don't ever mount just one of the devices in a RAID1 array. If you mount it r/w, then, even if you don't explicitly write anything to the device, it may get out of sync with the unmounted device, e.g. its filesystem journal has been updated. If you ever subsequently mount the other device, or the two of them together, your data will likely become corrupted. If you have to do this, make sure you mount your device r/o. Better yet, abandon the device you didn't mount. Zero out its RAID headers, and tell mdadm that that device has failed. Then you can if you like treat it as a new disk, which you can add as a replacement to your (now degraded) original RAID array.
A mirrored RAID array (level 1 or 5) protects you against hardware failure. It doesn't protect against rm -rf /, software errors, exploits, earthquakes, fire. Don't rely on RAID as a backup strategy.
Running a mirrored RAID only provides one line of defense against drive failures. It doesn't allow you to stop thinking about them. If a device in a RAID 1 starts failing and you aren't aware of it, your data will end up just as silently corrupted as it would be if you were running one drive. You have to watch your logs.

This document was updated for Alpine 2.4.6.

Loading needed modules

Start with loading the raid1 kernel module:

modprobe raid1

Add it to /etc/modules-load.d so it gets loaded during next reboot:

echo raid1 >> /etc/modules-load.d/raid1.conf

Creating the partitions

Please read up on partition types, and why you should consider using 0xda instead of 0xfd.

I will use /dev/sda and /dev/sdb in this document but your devices may be different. To find what disks you have available, look in /proc/partitions.

Create the partitions using fdisk.

fdisk /dev/sda

I will create one single partition of type Linux raid autodetect. Use n in fdisk to create the partition and t to set type. Logical volumes will be created later. My partition table looks like this ('p' prints the partition table):

   Device Boot      Start         End      Blocks  Id System
/dev/sda1               1       17753     8388261  fd Linux raid autodetect

Use w to write and quit. Do the same with your second disk.

fdisk /dev/sdb

Mine looks like this:

   Device Boot      Start         End      Blocks  Id System
/dev/sdb1               1       17753     8388261  fd Linux raid autodetect

Alternately, if your disks are the same size (as they should be, see above) you can copy the partition table from one to the other like this:

apk add sfdisk sfdisk -d /dev/sda | sfdisk /dev/sdb

Setting up the RAID array

Install mdadm to set up the arrays.

apk add mdadm

Create the array.

mdadm --create --level=1 --raid-devices=2 /dev/md0 /dev/sda1 /dev/sdb1

Monitoring sync status

You should now be able to see the array syncronize by looking at the contents of /proc/mdstat.

~ # cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 sdb1[1] sda1[0]
      8388160 blocks [2/2] [UU]
      [=========>...........]  resync = 45.3% (3800064/8388160) finish=0.3min  speed=200003K/sec

unused devices: <none>

You don't need to wait until it is fully syncronized to continue.

Saving config

Create the /etc/mdadm.conf file so mdadm knows how your RAID is set up:

mdadm --detail --scan > /etc/mdadm.conf

To make sure the raid devices start during the next reboot run:

rc-update add mdadm-raid

To use the raid array in /etc/fstab at boot, the mdadm service must be started at boot time:

rc-update add mdadm boot

rc-update add mdadm-raid boot

If you're not running Alpine from a hard disk install, use

lbu commit

as usual to save your configuration changes to your removable media.

The raid device /dev/md0 is now ready to be used with LVM or mkfs.

Adding a RAID after the installation

To add a softRAID to an already installed Alpine, you need to start with these 3 steps :

3. Loading needed modules

4. Creating the partitions

5. Setting up the RAID array

Then you have to update your initfs with the command mkinitfs.

1. Be sure /etc/mkinitfs/mkinitfs.conf contain raid which should look like this:

features="ata base ide scsi usb virtio ext4 lvm raid"

2. Update the initfs with this command

mkinitfs -c /etc/mkinitfs/mkinitfs.conf -b /

More Info on RAID

These resources may be helpful:

@@ Line 1: / Line 1: @@
-This document will show how to create hard disk mirroring using cheap IDE disks.
+[[Category:Storage]]
-This document was written for alpine-1.3.8 or later. It is tested with alpine-1.7.7.
+There are various forms of RAID: via a hardware RAID controller, "fake RAID", and "software RAID" using <code>mdadm</code>, which is Linux-only. These instructions discuss only the last form of RAID. Only RAID for arbitrary storage is covered here. It's possible to have one's system root {{Path|/}}, or {{Path|/var}}, or swap, or even one's {{Path|/boot}}, on a RAID array. See [[Setting up disks manually]] for more details about doing any of that.
-I will setup 1 raid device for use as physical storage for [[Setting up Logical Volumes with LVM | lvm]] .
+== RAID Levels ==
+There are several "levels" of RAID to choose between:
+* RAID0 essentially just glues two devices together, making a larger virtual drive. Reads and writes are "striped" between the drives for speed improvements. (That is, your hardware may read from, or write different data to, multiple devices in parallel.) A "device" here is usually a partition of a hard drive.
+* RAID1 "mirrors" writes to two devices, for improved safety. Then if one of the devices fails, the data will still be available on the other.
+* RAID5 is similar to RAID1, but it uses three devices and provides the space of two of them. The data will be preserved as long as any two of the three devices continue to work.
+There are other RAID levels as well. [https://www.acnc.com/raid/ Here is an explanation of their differences.]
+== Advice ==
+* Your {{Path|/boot}} partition should either not be on RAID, or else be on a RAID1 array, with no further layers of encryption or LVM. (Alpine's default bootloader extlinux can't handle either. Grub2 can handle {{Path|/boot}} being on LVM.) The usual practice is to create a small (32--100 MB) partition for {{Path|/boot}}. That can be a mirrored (RAID1) volume, however this is just for post-init access. That way, when you write a new kernel or bootloader config file to {{Path|/boot}}, it gets written to multiple physical partitions. During the pre-init, bootloader phase, only one of those partitions will be used (and it will be mounted read-only).
+* It's important to note that extlinux can't boot from partitions created with <code>mdadm</code> metadata version 1.2, which is the default. To be on the safe side, it's recommended to create the partition based on an older metadata version, with the <code>--metadata=0.90</code> parameter for the {{Path|/boot}} partition.
+* You can put swap on a RAID0 volume, but there doesn't seem to be any good reason to do so. The Linux kernel already knows how to stripe several swap partitions. So you can just devote multiple ordinary (non-RAID) partitions to swap, and get the same effect. The downside from doing either of these things is that when one of your disks fails, the system will go down. For better reliability, create a mirrored (RAID1) volume and put swap there. This will let your system keep running if one of the disks fails.
+* All partitions in a RAID array should be the same size.
+* Don't ever mount just one of the devices in a RAID1 array. If you mount it r/w, then, even if you don't explicitly write anything to the device, it may get out of sync with the unmounted device, e.g. its filesystem journal has been updated. If you ever subsequently mount the other device, or the two of them together, your data will likely become corrupted. If you have to do this, make sure you mount your device r/o. Better yet, abandon the device you didn't mount. '''Zero out its RAID headers''', and tell <code>mdadm</code> that that device has failed. Then you can if you like treat it as a new disk, which you can add as a replacement to your (now degraded) original RAID array.
+* A mirrored RAID array (level 1 or 5) protects you against hardware failure. It doesn't protect against <code>rm -rf /</code>, software errors, exploits, earthquakes, fire. Don't rely on RAID as a backup strategy.
+* Running a mirrored RAID only provides one line of defense against drive failures. It doesn't allow you to stop thinking about them. If a device in a RAID 1 starts failing and you aren't aware of it, your data will end up just as silently corrupted as it would be if you were running one drive. You have to watch your logs.
+This document was updated for Alpine 2.4.6.
 == Loading needed modules ==
-Start with loading the ide-disk and raid1 kernel modules. If you use SATA or SCSI disks you will not need the ide-disk module.
+Start with loading the raid1 kernel module:
- modprobe ide-disk
+{{Cmd|modprobe raid1}}
- modprobe raid1
-Add them to /etc/modules so they get loaded during next reboot.
+Add it to {{Path|/etc/modules-load.d}} so it gets loaded during next reboot:
- echo ide-disk >> /etc/modules
+{{Cmd|echo raid1 >> /etc/modules-load.d/raid1.conf}}
- echo raid1 >> /etc/modules
 == Creating the partitions ==
-I will use /dev/hde and /dev/hdg in this document but you will probably use /dev/hda and /dev/hdc. Note that the disks should not be connected on the same IDE bus (sharing the same IDE cable). To find what disks you have available, look in /proc/partitions or look at the /dev/disk* links that the mdev system has created.
- ls -l /dev/disk*
+Please read up on [https://raid.wiki.kernel.org/index.php/Partition_Types partition types], and why you should consider using 0xda instead of 0xfd.
- lrwxrwxrwx    1 root     root            3 Oct 17 13:23 /dev/disk -> hda
- lrwxrwxrwx    1 root     root            3 Oct 17 13:23 /dev/disk0 -> hda
+I will use {{Path|/dev/sda}} and {{Path|/dev/sdb}} in this document but your devices may be different. To find what disks you have available, look in {{Path|/proc/partitions}}.
- lrwxrwxrwx    1 root     root            3 Oct 17 13:23 /dev/disk1 -> hdc
 Create the partitions using fdisk.
- fdisk /dev/hda
+{{Cmd|fdisk /dev/sda}}
-I will create one single partition of type Linux raid autodetect. Use '''n''' in fdisk to create the partition and '''t''' to set type. Logical volumes will be created later. My partition table looks like this ('p' to print partition table):
+I will create one single partition of type Linux raid autodetect. Use '''n''' in fdisk to create the partition and '''t''' to set type. Logical volumes will be created later. My partition table looks like this ('p' prints the partition table):
      Device Boot      Start         End      Blocks  Id System
-  /dev/hda1               1       17753     8388261  fd Linux raid autodetect
+  /dev/sda1               1       17753     8388261  fd Linux raid autodetect
 Use '''w''' to '''w'''rite and quit.
 Do the same with your second disk.
- fdisk /dev/hdc
+{{Cmd|fdisk /dev/sdb}}
 Mine looks like this:
      Device Boot      Start         End      Blocks  Id System
-  /dev/hdc1               1       17753     8388261  fd Linux raid autodetect
+  /dev/sdb1               1       17753     8388261  fd Linux raid autodetect
+Alternately, if your disks are the same size (as they should be, see [[#Advice|above]]) you can copy the partition table from one to the other like this:
+{{Cmd|1=apk add sfdisk
+sfdisk -d /dev/sda {{!}} sfdisk /dev/sdb}}
-== Setting up the raid array ==
+== Setting up the RAID array ==
 Install mdadm to set up the arrays.
- apk_add mdadm
+{{Cmd|apk add mdadm}}
+Create the array.
-If you use an alpine version earlier than 1.7.3 you need to make the /dev nodes before creating the arrays.
+{{Cmd|1=mdadm --create --level=1 --raid-devices=2 /dev/md0 /dev/sda1 /dev/sdb1}}
- mknod /dev/md0 b 9 0
- mknod /dev/md1 b 9 1
-Create the arrays.
+== Monitoring sync status ==
- mdadm --create -l 1 -n 2 /dev/md0 /dev/hde1 /dev/hdg1
+You should now be able to see the array syncronize by looking at the contents of {{Path|/proc/mdstat}}.
- mdadm --create -l 1 -n 2 /dev/md1 /dev/hde2 /dev/hdg2
-== Monitorin sync status ==
+  ~ # cat /proc/mdstat
-You should now be able to see the array syncronize by looking at the contents of /proc/mdstat.
+  Personalities : [raid1]
-  ~ $ cat /proc/mdstat
+  md0 : active raid1 sdb1[1] sda1[0]
-  Personalities : [raid1]
+        8388160 blocks [2/2] [UU]
-  md1 : active raid1 hdg2[1] hde2[0]
+        [=========>...........]  resync = 45.3% (3800064/8388160) finish=0.3min  speed=200003K/sec
-        38582592 blocks [2/2] [UU]
-        [==>..................]  resync = 10.5% (4056192/38582592) finish=68.5min speed=8388K/sec
- md0 : active raid1 hdg1[1] hde1[0]
-blocks [2/2] [UU]
   unused devices: <none>
-You don't need to wait til it is fully syncronized to continue.
+You don't need to wait until it is fully syncronized to continue.
 == Saving config ==
-Create the /etc/mdadm.conf file so mdadm knows how your raid setup is:
+Create the /etc/mdadm.conf file so mdadm knows how your RAID is set up:
- mdadm --detail --scan > /etc/mdadm.conf
+{{Cmd|mdadm --detail --scan > /etc/mdadm.conf}}
 To make sure the raid devices start during the next reboot run:
- rc_add -s 10 -k mdadm-raid
+{{Cmd|rc-update add mdadm-raid}}
+To use the raid array in {{Path|/etc/fstab}} at boot, the mdadm service must be started at boot time:
+{{Cmd|rc-update add mdadm boot}}
+{{Cmd|rc-update add mdadm-raid boot}}
+If you're not running Alpine from a hard disk install, use {{Cmd|lbu commit}} as usual to save your configuration changes to your removable media.
+The raid device {{Path|/dev/md0}} is now ready to be used with [[Setting up Logical Volumes with LVM|LVM]] or mkfs.
+== Adding a RAID after the installation ==
+To add a softRAID to an already installed Alpine, you need to start with these 3 steps :
+. [[Setting_up_a_software_RAID_array#Loading_needed_modules|Loading needed modules]]
+. [[Setting_up_a_software_RAID_array#Creating_the_partitions|Creating the partitions]]
+. [[Setting_up_a_software_RAID_array#Setting_up_the_RAID_array|Setting up the RAID array]]
+Then you have to update your initfs with the command mkinitfs.
+. Be sure {{Path|/etc/mkinitfs/mkinitfs.conf}} contain '''raid''' which should look like this:
+<code>features="ata base ide scsi usb virtio ext4 lvm raid"</code>
+. Update the initfs with this command
+{{Cmd|mkinitfs -c /etc/mkinitfs/mkinitfs.conf -b /}}
-The ''-s 10'' option is to make sure that the raid arrays are started early, before things like lvm and localmount.
+== More Info on RAID ==
+These resources may be helpful:
-Use lbu commit as usual to save configs to usb or floppy.
+* [https://wiki.archlinux.org/index.php/RAID Arch wiki page on RAID]
+* [https://wiki.archlinux.org/index.php/Software_RAID_and_LVM Arch wiki page on RAID and LVM]
+* [https://wiki.archlinux.org/index.php/Convert_a_single_drive_system_to_RAID Arch wiki page on Converting an existing system to RAID] [https://web.archive.org/web/20120420141131/https://en.gentoo-wiki.com/wiki/Migrate_to_RAID Gentoo wiki page on the same]
+* [https://web.archive.org/web/20130419074430/http://en.gentoo-wiki.com/wiki/RAID/Software Gentoo Linux Wiki: RAID/Software (via archive.org)]
+* [https://web.archive.org/web/20130325021303/http://en.gentoo-wiki.com/wiki/Software_RAID_Install Gentoo Linux Wiki: Software RAID Install (via archive.org)]
+* https://www.gentoo.org/doc/en/gentoo-x86-tipsntricks.xml#software-raid
+* [https://web.archive.org/web/20150223050854/http://www.gentoo.org/doc/en/gentoo-x86+raid+lvm2-quickinstall.xml Gentoo Linux x86 with Software Raid and LVM2 Quick Install Guide (via archive.org)]
-You should now be able to create swap on /dev/md0 and a filesystem on /dev/md1 using the ''mkswap'' and ''mkfs.*'' commands.
+* [https://web.archive.org/web/20120919051801/http://yannickloth.be/blog/2010/08/01/installing-archlinux-with-software-raid1-encrypted-filesystem-and-lvm2/ Yannick Loth: Installing Archlinux with software raid1, encrypted filesystem and LVM2 (via archive.org)]
+* [https://web.archive.org/web/20171125120354/https://anonscm.debian.org/gitweb/?p=pkg-mdadm/mdadm.git;a=blob_plain;f=debian/FAQ;hb=HEAD Debian MADM FAQ (via archive.org)]
+* [https://tldp.org/FAQ/Linux-RAID-FAQ/x37.html Linux Documentation Project: Linux RAID FAQ]
+* [https://web.archive.org/web/20150214220324/http://linux-101.org/howto/arch-linux-software-raid-installation-guide Linux 101: Arch Linux software RAID installation guide (via archive.org)]
+* https://raid.wiki.kernel.org/index.php/Linux_Raid