Stage 3 · Build
Storage Reliability & Performance
Filesystem Repair & Recovery
fsck, xfs_repair, journal replay, and snapshot-based recovery workflows.
fsck Basics
fsck (filesystem check) verifies and repairs filesystem inconsistencies. It must be run on unmounted filesystems — running fsck on a mounted filesystem causes data loss.
# Check filesystem (dry run)
sudo fsck -n /dev/sda1
# e2fsck 1.46.5 (15-Mar-2021)
# /dev/sda1: clean, 234567/6553600 files, 12345678/52428800 blocks
# Repair automatically
sudo fsck -y /dev/sda1
# Repair with backup
sudo fsck -b 8192 /dev/sda1
# -b uses a backup superblock
# Check specific filesystem type
sudo fsck.ext4 /dev/sda1
# Force check (even if clean)
sudo fsck -f /dev/sda1fsck can only repair unmounted filesystems. For root filesystems, boot from rescue media or use fsck at boot (initramfs).
# Full ext4 check with options
sudo e2fsck -f -v /dev/sda1
# Pass 1: Checking inodes, blocks, and sizes
# Pass 2: Checking directory structure
# Pass 3: Checking directory connectivity
# Pass 4: Checking reference counts
# Pass 5: Checking group summary information
# Common errors fixed:
# - Duplicate blocks
# - Orphaned inodes
# - Directory entry pointing to wrong inode
# - Missing blocks in use map
# If root filesystem, force at boot
sudo tune2fs -C 50 /dev/sda1 # Set mount count to force check
sudo tune2fs -I 30 /dev/sda1 # Set interval to 30 daysext4 fsck has 5 passes. Pass 1 checks inodes, Pass 2 checks directory structure, Pass 3 checks connectivity, Pass 4 checks reference counts, Pass 5 checks group summaries.
xfs_repair
# Check XFS filesystem (dry run)
sudo xfs_repair -n /dev/sda1
# Phase 1 - find and verify superblock
# Phase 2 - using primary superblock
# Phase 3 - checking free space
# Phase 4 - checking contents (sounds like the scariest phase)
# Repair XFS filesystem
sudo xfs_repair /dev/sda1
# Repair with log clear (if log is corrupt)
sudo xfs_repair -L /dev/sda1
# WARNING: This clears the journal, may lose recent data
# Check with log replay
sudo xfs_repair /dev/sda1 # Automatically replays log
# Repair from live CD
sudo xfs_repair /dev/sda1xfs_repair replays the journal by default. If the journal is corrupt, use -L to clear it. This may lose the most recent transactions.
Journal Replay
Both ext4 and XFS journal replay happens automatically at mount time. The filesystem checks the journal and replays any uncommitted transactions. This ensures consistency after an unclean shutdown.
# ext4: force journal replay
sudo tune2fs -O has_journal /dev/sda1
sudo mount /dev/sda1 /mnt # Journal replays automatically
# Check journal status
sudo tune2fs -l /dev/sda1 | grep journal
# Journal inode: 8
# Journal backup: blocks
# XFS: journal is internal by default
# Replay happens automatically at mount
# If mount fails due to journal corruption:
# ext4: sudo e2fsck -y /dev/sda1
# XFS: sudo xfs_repair -L /dev/sda1Journal replay is automatic and safe. If mount fails, the journal may be corrupt — run fsck to repair it.
Snapshot-Based Recovery
# Btrfs: restore from snapshot
sudo btrfs subvolume delete /mnt/corrupted_data
sudo btrfs subvolume snapshot /mnt/.snapshots/good_copy /mnt/corrupted_data
# ZFS: rollback to snapshot
sudo zpool import mypool
sudo zfs rollback mypool/data@last_known_good
# LVM: restore from snapshot
sudo lvmerge /dev/myvg/data_snap /dev/myvg/data
# For databases, prefer point-in-time recovery
# Use the database's WAL/binlog for consistencySnapshot recovery is instant and consistent. For databases, combine filesystem snapshots with database WAL/binlog for point-in-time recovery.
Boot from Rescue Media
# Boot from Ubuntu/Debian live USB
# Select "Try Ubuntu"
# Find the root partition
lsblk
# /dev/sda2 is the root partition
# Run fsck
sudo fsck -y /dev/sda2
# For XFS
sudo xfs_repair /dev/sda2
# If GRUB is broken
sudo mount /dev/sda2 /mnt
sudo mount /dev/sda1 /mnt/boot
for i in /dev /proc /sys /run; do sudo mount --bind $i /mnt$i; done
sudo chroot /mnt
grub-install /dev/sda
update-grub
exit
sudo umount -R /mntBoot from live media when the root filesystem is corrupted or GRUB is broken. Always run fsck before attempting to repair GRUB.
Prevention Strategies
- Use UPS to prevent unclean shutdowns
- Enable write barriers (default) for data safety
- Use snapshots for point-in-time recovery
- Schedule regular scrubs (Btrfs, ZFS, mdadm)
- Monitor SMART for disk health
- Keep filesystem tools updated
- Test recovery procedures regularly
Run btrfs scrub start /data weekly and zpool scrub mypool monthly. Scrubs verify data integrity and repair silent corruption using parity.
Running fsck on a mounted filesystem can cause catastrophic data loss. Always unmount first, or boot from rescue media for root filesystems.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.