Stage 3 · Build
Linux Kernel Internals
Interrupts & Softirqs
Hardware interrupts, softirqs, tasklets, and NAPI for networking — how the kernel handles asynchronous events.
Hardware Interrupts
When a hardware device needs attention — a packet arrives, a disk I/O completes, a key is pressed — it sends an interrupt signal to the CPU. The CPU pauses its current work, saves its state, and jumps to an interrupt handler registered for that IRQ line.
# System-wide interrupt counts
cat /proc/interrupts | head -10
# CPU0 CPU1 CPU2 CPU3
# 23: 123456 134567 145678 156789 IO-APIC 23-edge ehci_hcd
# 42: 0 0 0 1234567 PCI-MSI 524289-edge nvme0q3
# Per-CPU interrupt distribution
cat /proc/interrupts | awk '{print $1, $2+$3+$4+$5}' | sort -nrk 2 | head -10Each interrupt line shows the total count per CPU. A high count on a single CPU indicates the interrupt is not balanced across cores.
Interrupt Handlers
Interrupt handlers (also called Interrupt Service Routines or ISRs) must be fast and cannot sleep. They run with interrupts disabled on the local CPU. Heavy processing is deferred to softirqs or tasklets, which run later with interrupts enabled.
- Top half — The actual ISR. Runs immediately, handles urgent work, cannot sleep
- Bottom half — Deferred work. Runs later via softirqs, tasklets, or workqueues
- Hardirq context — Interrupt handlers run here. No scheduling, no sleeping
- Softirq context — Bottom halves run here. Can be preempted by hardirqs
cat /proc/softirqs
# CPU0 CPU1 CPU2 CPU3
# HI: 12345 12345 12345 12345
# TIMER: 1234567 1234567 1234567 1234567
# NET_TX: 1234 1234 1234 1234
# NET_RX: 2345678 2345678 2345678 2345678
# BLOCK: 3456789 3456789 3456789 3456789
# IRQ_POLL: 0 0 0 0
# TASKLET: 567890 567890 567890 567890
# SCHED: 4567890 4567890 4567890 4567890NET_RX softirqs are the most frequent for network-heavy servers. If they are concentrated on one CPU, consider enabling RPS (Receive Packet Steering).
Softirqs
Softirqs are statically allocated kernel threads that handle deferred work. There are 10 predefined softirq types in the kernel. They run at the end of every interrupt handler or when ksoftirqd is scheduled.
| Softirq | Purpose |
|---|---|
| HI | High-priority tasklets |
| TIMER | Kernel timers and delays |
| NET_TX | Network packet transmission |
| NET_RX | Network packet reception |
| BLOCK | Block device completions |
| IRQ_POLL | Interrupt-driven polling |
| TASKLET | Tasklet execution |
| SCHED | Scheduler load balancing |
| HRTIMER | High-resolution timers |
| RCU | Read-Copy-Update synchronization |
Tasklets
Tasklets are built on top of softirqs. They provide a simpler interface for deferred work and are serialized — the same tasklet never runs on two CPUs simultaneously. This makes them safe for driver use without explicit locking.
# Tasklet statistics (kernel debug feature)
cat /proc/softirqs | grep TASKLET
# In practice, you see tasklet impact through other metrics
# High NET_RX with single-CPU concentration suggests
# tasklet serialization bottleneckModern kernels prefer workqueues over tasklets for most use cases. Workqueues run in process context and can sleep, making them more flexible.
NAPI for Networking
NAPI (New API) is the kernel's mechanism for high-performance network packet processing. Instead of generating an interrupt for every packet, NAPI switches to polling mode under high load. The kernel polls the NIC for packets in batches, reducing interrupt overhead dramatically.
# Check NIC interrupt coalescence settings
ethtool -c eth0
# rx-usecs: 3
# rx-frames: 0
# rx-usecs-irq: 0
# adaptive-rx: on
# Adjust coalescing for throughput vs latency
ethtool -C eth0 rx-usecs 50 rx-frames 25 # Higher throughput
ethtool -C eth0 rx-usecs 0 rx-frames 1 # Lower latency
# Check NAPI busy poll settings
sysctl net.core.busy_read
sysnet.core.busy_pollHigher coalescing values reduce CPU usage but increase latency. Tune based on your workload: low latency for trading systems, high throughput for bulk data transfer.
Interrupt Affinity
Interrupt affinity controls which CPU handles which interrupt. Spreading interrupts across CPUs prevents any single core from becoming a bottleneck. This is critical for multi-queue NICs where each queue has its own interrupt.
# View current affinity
cat /proc/irq/42/smp_affinity
# 1 (CPU 0 only)
# Set affinity to CPU 1
echo 2 > /proc/irq/42/smp_affinity
# Set affinity to CPUs 0-3
echo f > /proc/irq/42/smp_affinality
# Use irqbalance for automatic distribution (if available)
systemctl status irqbalance
# Multi-queue NIC: check per-queue interrupts
cat /proc/interrupts | grep nvmeFor NUMA systems, always set interrupt affinity to CPUs on the same NUMA node as the NIC. Cross-NUMA interrupts add significant latency.
irqbalance automatically distributes interrupts across CPUs. For high-performance networking, manual tuning with set_irq_affinity gives better results. Disable irqbalance and set affinities explicitly for predictable performance.
A misconfigured device or broken driver can generate thousands of interrupts per second on a single CPU, freezing the system. Check /proc/interrupts if the system becomes unresponsive. A single CPU at 100% with high interrupt counts is a classic sign.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.