Stage 3 · Build
Storage & Filesystems
The Block Layer
Request queues, I/O schedulers (bfq, kyber, none), and blk-mq — how Linux manages disk I/O.
Block Layer Overview
The block layer sits between the filesystem and the disk driver. It handles I/O scheduling, request merging, and dispatching to the hardware. Understanding the block layer is essential for storage performance tuning.
# List block devices
lsblk
# NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
# sda 8:0 0 500G 0 disk
# ├─sda1 8:1 0 1G 0 part /boot
# └─sda2 8:2 0 499G 0 part
# └─vg0-root 253:0 0 499G 0 lvm /
# View I/O statistics
iostat -xz 1
# Device r/s w/s rkB/s wkB/s rrqm/s wrqm/s %util
# sda 12.00 45.00 480.00 180.00 2.00 10.00 35.00%util near 100% indicates the disk is saturated. High rkB/s and wkB/s with low r/s indicates large sequential I/O.
Request Queues
Each block device has a request queue that holds pending I/O operations. The queue depth determines how many requests can be in flight simultaneously. NVMe devices support much deeper queues than SATA devices.
# View queue parameters for a device
cat /sys/block/sda/queue/nr_requests
# 256
cat /sys/block/sda/queue/scheduler
# [mq-deadline] kyber bfq none
# Queue depth
cat /sys/block/sda/queue/max_sectors_kb
# 512
# NVMe queue depth
cat /sys/block/nvme0n1/queue/nr_requests
# 1023nr_requests controls the maximum requests in the queue per priority level. Increase it for workloads with many concurrent I/O operations.
I/O Schedulers
I/O schedulers reorder and merge requests to improve performance. Modern kernels use multi-queue schedulers designed for fast storage devices.
| Scheduler | Strategy | Best For |
|---|---|---|
| none | No reordering, direct dispatch | NVMe, fast SSDs |
| mq-deadline | Deadline-based ordering | General purpose, databases |
| kyber | Two queues (read/write) | Fast storage, low latency |
| bfq | Fair queuing, budget-based | Desktop, interactive workloads |
blk-mq: Multi-Queue Block Layer
blk-mq (block multi-queue) is the modern block layer. It uses per-CPU request queues to eliminate lock contention on multi-core systems. Each CPU has its own queue, and requests are dispatched directly to the hardware queue.
# Check if blk-mq is active
cat /sys/block/sda/queue/mode
# 0 = none (blk-mq is active)
# View hardware queues
ls /sys/block/sda/mq/
# 0 1 2 3 4 5 6 7
# Check per-CPU queue dispatch
cat /sys/block/sda/mq/*/cpu_list
# 0-0
# 1-1
# 2-2
# 3-3blk-mq eliminates the single bottleneck of the legacy block layer. Each CPU dispatches I/O to its own queue, and the hardware handles queue arbitration.
Choosing a Scheduler
# Change scheduler at runtime
echo kyber > /sys/block/nvme0n1/queue/scheduler
# Persistent via udev rule
cat > /etc/udev/rules.d/60-ioscheduler.rules << 'EOF'
ACTION=="add|change", KERNEL=="nvme*", ATTR{queue/scheduler}="none"
ACTION=="add|change", KERNEL=="sd*", ATTR{queue/scheduler}="mq-deadline"
ACTION=="add|change", KERNEL=="vd*", ATTR{queue/scheduler}="none"
EOF
# Test with fio
fio --name=test --ioengine=libaio --direct=1 --bs=4k --iodepth=32 --rw=randread --size=1G --filename=/dev/sdaNVMe devices should use 'none' (no scheduler overhead). SATA SSDs benefit from 'mq-deadline'. HDDs benefit from 'bfq' for interactive fairness.
Block Layer Tuning
# Read-ahead buffer (KB)
cat /sys/block/sda/queue/read_ahead_kb
# 128
# Increase for sequential workloads
echo 2048 > /sys/block/sda/queue/read_ahead_kb
# Maximum number of requests per queue
echo 1024 > /sys/block/sda/queue/nr_requests
# Max sectors per request (KB)
cat /sys/block/sda/queue/max_sectors_kb
# 512
# Rotational media flag (0 for SSD, 1 for HDD)
cat /sys/block/sda/queue/rotational
# 0Read-ahead predicts sequential access patterns and prefetches data. Increase it for large sequential reads (backups, video). Decrease it for random access (databases).
Before investigating storage performance issues, always check the scheduler with cat /sys/block/<dev>/queue/scheduler. An inappropriate scheduler can cause significant latency spikes.
Changing the scheduler affects all processes using that device. Test changes in staging first. For production, use udev rules to apply consistent scheduler settings at boot.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.