Golden Signals

What Are Golden Signals?

The four golden signals, defined by Google's SRE book, are latency, traffic, errors, and saturation. They provide a framework for measuring service health. Every service dashboard should visualize these four signals prominently.

Latency

Latency measures the time taken to serve a request. Separate successful requests from failed ones because failed requests may be fast (immediate error responses). Use histogram quantiles to show the distribution.

# P50 latency
histogram_quantile(0.50, sum by(le) (rate(http_request_duration_seconds_bucket{status!~"5.."}[5m])))

# P95 latency
histogram_quantile(0.95, sum by(le) (rate(http_request_duration_seconds_bucket{status!~"5.."}[5m])))

# P99 latency
histogram_quantile(0.99, sum by(le) (rate(http_request_duration_seconds_bucket{status!~"5.."}[5m])))

Traffic

Traffic measures the demand on your service. It is typically requests per second but can be transactions, messages, or any unit of work. Traffic helps you understand load patterns and plan capacity.

# Total request rate
sum(rate(http_requests_total[5m]))

# Request rate by endpoint
sum by(endpoint) (rate(http_requests_total[5m]))

# Request rate by method
sum by(method) (rate(http_requests_total[5m]))

Errors

Errors measure the rate of failed requests. Distinguish between different error types: 4xx client errors, 5xx server errors, timeouts, and application errors. The error rate is often the most critical signal for alerting.

# Error ratio (5xx / total)
sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m]))

# Error rate per endpoint
sum by(endpoint) (rate(http_requests_total{status=~"5.."}[5m]))
  / sum by(endpoint) (rate(http_requests_total[5m]))

Saturation

Saturation measures how full your service is. It represents the level of resource utilization that limits service capacity. Choose the resource that is most likely to be exhausted first.

# CPU saturation
100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Memory saturation
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100

# Goroutine saturation (for Go services)
go_goroutines

Apdex Score

Apdex is a standardized measure of user satisfaction. It classifies requests into satisfied, tolerating, and frustrated buckets based on response time thresholds. An Apdex score ranges from 0 (no users satisfied) to 1 (all users satisfied).

# Apdex with 200ms satisfied, 800ms tolerating
(
  sum(rate(http_request_duration_seconds_bucket{le="0.2"}[5m]))
  + sum(rate(http_request_duration_seconds_bucket{le="0.8"}[5m]))
) / 2
/ sum(rate(http_request_duration_seconds_count[5m]))

The Four Golden Signals Dashboard

• Top row: Stat panels showing current latency P95, total QPS, error ratio, and CPU saturation.
• Second row: Time series for latency percentiles (P50, P95, P99) over time.
• Third row: Time series for traffic (total QPS and per-endpoint breakdown).
• Fourth row: Time series for error ratio and error count.
• Bottom row: Time series for resource saturation (CPU, memory, goroutines).