Stage 3 · Build
Storage Reliability & Performance
RAID with mdadm
RAID levels, rebuild behavior, write holes, and monitoring /proc/mdstat safely.
RAID Levels
mdadm supports multiple RAID levels for different combinations of performance, redundancy, and capacity. Each level has trade-offs that determine where it should be used.
| Level | Min Disks | Fault Tolerance | Use Case |
|---|---|---|---|
| RAID 0 | 2 | None | Performance only (no redundancy) |
| RAID 1 | 2 | N-1 disks | Mirrors, boot drives |
| RAID 5 | 3 | 1 disk | General purpose, read-heavy |
| RAID 6 | 4 | 2 disks | High redundancy, large arrays |
| RAID 10 | 4 | N/2 disks | Performance + redundancy |
Creating RAID Arrays
# Create RAID 1 mirror
sudo mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb /dev/sdc
# Create RAID 5
sudo mdadm --create /dev/md0 --level=5 --raid-devices=4 /dev/sd{b,c,d,e}
# Create RAID 10
sudo mdadm --create /dev/md0 --level=10 --raid-devices=4 /dev/sd{b,c,d,e}
# Save configuration
sudo mdadm --detail --scan >> /etc/mdadm/mdadm.conf
sudo update-initramfs -u
# View array details
cat /proc/mdstat
# md0 : active raid1 sdc[1] sdb[0]
# 1953513472 blocks super 1.2 [2/2] [UU]
sudo mdadm --detail /dev/md0mdadm creates software RAID arrays at the block device level. The [UU] in /proc/mdstat means both drives are up. [U_] means one drive has failed.
Monitoring /proc/mdstat
# Quick status check
cat /proc/mdstat
# md0 : active raid1 sdc[1] sdb[0]
# 1953513472 blocks super 1.2 [2/2] [UU]
# [>....................] recovery = 3.2% (62500000/1953513472) finish=45.2min speed=500000K/sec
# Detailed status
sudo mdadm --detail /dev/md0
# /dev/md0
# Version : 1.2
# Creation Time : Mon Jan 15 10:30:00 2024
# Raid Level : raid1
# Array Size : 1953513472 (1863.01 GiB 2000.40 GB)
# Raid Devices : 2
# Avail Dev Size : 1953513472 (1863.01 GiB 2000.40 GB)
# State : clean
# Active Devices : 2
# Working Devices : 2
# Failed Devices : 0
# Monitor health
sudo mdadm --test /dev/md0Check /proc/mdstat regularly. A degraded array ([_U] or [U_]) means a disk has failed. Recovery shows rebuild progress.
Rebuild Behavior
# Check current rebuild speed limit
cat /proc/sys/dev/raid/speed_limit_min
# 100000 (KB/s)
cat /proc/sys/dev/raid/speed_limit_max
# 200000 (KB/s)
# Increase rebuild speed (more I/O, faster rebuild)
echo 500000 > /proc/sys/dev/raid/speed_limit_min
echo 1000000 > /proc/sys/dev/raid/speed_limit_max
# Set rebuild priority in mdadm
echo 1024 > /proc/sys/dev/raid/md_stripe_cache_size
# Monitor rebuild progress
watch -n 5 cat /proc/mdstatRebuild speed defaults to conservative values to avoid impacting production I/O. Increase it during maintenance windows for faster recovery.
The Write Hole
The write hole is a classic RAID 5/6 problem. During a write, if power fails after writing parity but before writing data (or vice versa), the parity becomes inconsistent. On RAID 5, this can cause silent data corruption during a subsequent rebuild.
# Enable write-intent bitmap (reduces rebuild time and write hole risk)
sudo mdadm --create /dev/md0 --level=5 --raid-devices=4 \
--bitmap=internal /dev/sd{b,c,d,e}
# Add bitmap to existing array
sudo mdadm --bitmap=internal /dev/md0
# Check bitmap status
sudo mdadm --detail /dev/md0 | grep bitmap
# Bitmap: 1953513472 blocks, 65536 chunks, file bitmap
# For RAID 6, consider write journal (mdadm 4.2+)
# RAID 6 is inherently safer than RAID 5Write-intent bitmaps track which blocks are being written, allowing the array to resync only changed blocks after a crash. This dramatically reduces rebuild time.
Failure Recovery
# Mark a disk as failed
sudo mdadm --fail /dev/md0 /dev/sdb
# Remove the failed disk
sudo mdadm --remove /dev/md0 /dev/sdb
# Add a new disk
sudo mdadm --add /dev/md0 /dev/sdf
# Monitor rebuild
cat /proc/mdstat
# md0 : active raid1 sdf[2] sdc[1]
# 1953513472 blocks super 1.2 [2/1] [_U]
# [>....................] recovery = 3.2% ...
# Verify array health
sudo mdadm --detail /dev/md0mdadm handles hot-swapping gracefully. The array runs degraded during replacement, then rebuilds onto the new disk. Monitor until rebuild completes.
Configure mdmonitor for email alerts on array events. Enable it with sudo systemctl enable mdmonitor. Check /etc/mdadm/mdadm.conf for the mail address.
With large drives (4TB+), RAID 5 rebuild times can exceed 24 hours. During rebuild, a second failure causes data loss. Use RAID 6 or RAID 10 for large drives.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.