Stage 3 · Build

Storage & Filesystems

NVMe & Storage Performance

Queue depth, namespaces, and optimizing for flash media — getting the most from NVMe drives.

6 min readLinux Deep DivesBuild

NVMe Overview

NVMe (Non-Volatile Memory Express) is a storage protocol designed specifically for flash media. Unlike AHCI (designed for spinning disks), NVMe uses the PCIe bus directly, eliminating the SATA bottleneck. NVMe drives achieve millions of IOPS with microseconds of latency.

Feature	SATA/AHCI	NVMe/PCIe
Max throughput	600 MB/s	7 GB/s (PCIe 4.0)
Max IOPS	100K	1M+
Queue depth	32	65535
Queues	1	64K
Latency	100+ us	10-20 us
Protocol	AHCI	NVMe
Interface	SATA	PCIe

NVMe Queues

NVMe supports up to 65,535 I/O queues, each with up to 65,535 entries. This massive parallelism eliminates the single-queue bottleneck of SATA. Each CPU core can have its own queue, enabling lock-free I/O at scale.

NVMe queue configuration


# Check NVMe queue count
cat /sys/block/nvme0n1/queue/nr_queues
# 32

# Check queue depth
cat /sys/block/nvme0n1/queue/nr_requests
# 1023

# View NVMe controller info
sudo nvme id-ctrl /dev/nvme0 | head -20
# vid     : 0x144d
# ssvid   : 0x144d
# sn      : S5EVNX0R123456
# mn      : Samsung SSD 970 EVO Plus 1TB

# Number of queues supported
sudo nvme id-ctrl /dev/nvme0 | grep "^sq"

NVMe queues are mapped to CPU cores by the driver. More queues mean more parallelism. The default is usually auto-detected based on CPU count.

NVMe Namespaces

A namespace is a storage region that can be formatted with a filesystem. Each NVMe drive can have multiple namespaces, each with its own logical block addressing. This allows partitioning at the hardware level.

List NVMe namespaces


# List namespaces on a drive
sudo nvme list-ns /dev/nvme0
# ID     NSIZE       USE
# 1      1024GiB     1024GiB

# Create a new namespace
sudo nvme create-ns /dev/nvme0 -s 512 -c 512

# Format a namespace
sudo nvme format /dev/nvme0n2 -l 1

# Delete a namespace
sudo nvme delete-ns /dev/nvme0n2 -c

Namespaces are useful for isolating storage at the hardware level. Each namespace can have different block sizes and protection information.

NVMe Tuning

NVMe performance tuning


# Set I/O scheduler to none (no scheduler overhead)
echo none > /sys/block/nvme0n1/queue/scheduler

# Increase queue depth
echo 1024 > /sys/block/nvme0n1/queue/nr_requests

# Set interrupt affinity for NUMA
cat /sys/block/nvme0n1/device/numa_node
# 0
echo 1 > /proc/irq/<nvme_irq>/smp_affinity

# Disable power management (for consistent performance)
echo 0 > /sys/block/nvme0n1/device/power/control

# Check power state
sudo nvme get-feature /dev/nvme0 -f 0x02 -H

Disable power management for latency-sensitive workloads. Power state transitions add microseconds of latency that matter for databases and real-time applications.

NVMe Monitoring

Monitor NVMe health


# SMART health log
sudo nvme smart-log /dev/nvme0
# SMART/Health Information:
# Critical Warning:   0x00
# Temperature:        35 C
# Available Spare:    100%
# Available Spare Threshold: 10%
# Percentage Used:    2%
# Data Units Read:    12345678
# Data Units Written: 98765432
# Power Cycles:       1234
# Power On Hours:     5678
# Unsafe Shutdowns:   0

# Temperature threshold
sudo nvme get-feature /dev/nvme0 -f 0x04 -H

# Error log
sudo nvme error-log /dev/nvme0 -l 10

Monitor Percentage Used — when it reaches 100%, the drive warranty is exceeded. Available Spare shows remaining reserve blocks.

nvme-cli Tools

Essential nvme-cli commands


# List all NVMe drives
sudo nvme list
# Node   SN                   Model
# /dev/nvme0n1  S5EVNX0R123456  Samsung SSD 970 EVO Plus 1TB

# Firmware update
sudo nvme fw-download /dev/nvme0 -f firmware.bin
sudo nvme fw-activate /dev/nvme0 -s 1 -a 1

# Secure erase (CAUTION: destroys all data)
sudo nvme format /dev/nvme0 -s 2

# Self-test
sudo nvme device-self-test /dev/nvme0 -s 1
# 1 = short test, 2 = extended test

Always back up data before firmware updates or secure erases. Extended self-tests can take hours for large drives.

Use NVMe for databases

NVMe drives provide the lowest latency and highest IOPS available. For databases, use NVMe with no scheduler (none), deep queues, and power management disabled for consistent performance.

NVMe drives run hot under load

Monitor NVMe temperatures with nvme smart-log. Most drives throttle at 70C and shut down at 80C. Ensure adequate cooling for sustained write workloads.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.