Stage 6 · Operate
SLOs & Production Observability
RED and USE Methods
Applying request rate, errors, duration, utilization, saturation, and errors to services and resources.
RED Method
The RED method, coined by Tom Wilkie, focuses on user-facing services. It measures Rate, Errors, and Duration for every service. RED is the foundation of the golden signals approach and is best applied to request-driven services.
# Rate — requests per second
sum(rate(http_requests_total{job="api"}[5m]))
# Errors — error ratio
sum(rate(http_requests_total{job="api", status=~"5.."}[5m]))
/ sum(rate(http_requests_total{job="api"}[5m]))
# Duration — P99 latency
histogram_quantile(0.99,
sum by(le) (rate(http_request_duration_seconds_bucket{job="api"}[5m]))
)USE Method
The USE method, also by Tom Wilkie, focuses on infrastructure resources. It measures Utilization, Saturation, and Errors for every resource. USE is best applied to CPUs, memory, disks, and network interfaces.
# Utilization — CPU busy percentage
100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
# Saturation — memory pressure
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100
# Errors — disk errors
rate(node_disk_errors_total[5m])Applying RED to Services
Apply RED to every request-driven service. Each microservice, API endpoint, or gRPC service should expose rate, errors, and duration. This gives you consistent observability across your service mesh.
| Signal | Metric Type | Prometheus Metric |
|---|---|---|
| Rate | Counter | http_requests_total |
| Errors | Counter | http_requests_total{status=~"5.."} |
| Duration | Histogram | http_request_duration_seconds |
Applying USE to Resources
| Resource | Utilization | Saturation | Errors |
|---|---|---|---|
| CPU | cpu_busy | load_avg, runqueue | machine_checks |
| Memory | mem_used/mem_total | swap_used, OOM kills | correctable errors |
| Disk | disk_busy | io_queue_length | sector_errors |
| Network | bandwidth_used | interface_queue | dropped_packets |
When to Use Each
| Method | Focus | Best For | Example |
|---|---|---|---|
| RED | Services | APIs, microservices | Payment service |
| USE | Resources | Infrastructure | Database server |
Apply RED to services and USE to the infrastructure they run on. When a service shows high latency (RED), check the underlying resource saturation (USE). The combination gives you the full picture.
# RED: Service-level metrics
rate(http_requests_total{job="api"}[5m])
histogram_quantile(0.99, sum by(le) (rate(http_request_duration_seconds_bucket[5m])))
# USE: Infrastructure-level metrics
100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.