Aggregation Operators

Grouping with sum by, avg without, topk, count_values, vector matching, and label joins.

8 min readObservability & Monitoring StackOperate

Aggregation Basics

Aggregation operators combine multiple time series into fewer series. They reduce the number of results by grouping series based on label values. Prometheus provides 12 aggregation operators, each suited to different reduction patterns.

Operator	Purpose	Example Use
sum	Add values	Total QPS across all instances
avg	Average values	Mean latency across pods
min	Smallest value	Minimum available memory
max	Largest value	Maximum CPU utilization
count	Count series	Number of active targets
topk	Top K values	Top 5 endpoints by traffic

by and without Clauses

The by clause specifies which labels to keep when aggregating. The without clause specifies which labels to remove. They control the grouping dimension. The choice between them depends on which labels are more natural to list.

by vs without


# Sum by method — keep only the method label
sum by(method) (rate(http_requests_total[5m]))

# Same result using without — drop instance and job
sum without(instance, job) (rate(http_requests_total[5m]))

Prefer by over without

Using by is usually clearer because you explicitly state which labels matter. without can accidentally drop labels you intended to keep, especially when new labels are added to metrics later.

Common Aggregation Operators

Aggregation examples


# Count how many instances are reporting
count(up == 1)

# Top 3 endpoints by request rate
topk(3, sum by(endpoint) (rate(http_requests_total[5m])))

# Bottom 2 endpoints by request rate
bottomk(2, sum by(endpoint) (rate(http_requests_total[5m])))

# Count values of a label
count_values("version", kubernetes_pod_labels{label_app="myapp"})

# Standard deviation of latency
stddev(rate(http_request_duration_seconds_sum[5m]))

Vector Matching

When combining two vectors with binary operators, Prometheus must match series between them. By default, it uses one-to-one matching where both sides must have exactly one matching series. The on and ignoring modifiers control which labels participate in matching.

Vector matching modifiers


# One-to-one: match on endpoint only
rate(http_requests_total[5m]) / http_request_duration_seconds_count

# Using on to restrict matching labels
vector_a / on(endpoint) vector_b

# Ignoring specific labels for matching
vector_a / ignoring(instance, job) vector_b

Label Joins with group_left

group_left and group_right enable many-to-one matching. They let you attach labels from one vector to many series in another. This is useful for enriching metrics with metadata from a separate time series.

Label join example


# Attach team label from node labels to CPU metrics
rate(node_cpu_seconds_total[5m])
  * on(instance) group_left(team)
  node_info

Many-to-one can produce errors

If the many side has more than one matching series per one side value, you get a many-to-many error. Ensure the one side has unique label combinations. Use group_right when the many side is on the left.

Practical Examples

Real-world aggregation patterns


# Error ratio by endpoint
sum by(endpoint) (rate(http_requests_total{status=~"5.."}[5m]))
  /
sum by(endpoint) (rate(http_requests_total[5m]))

# P50 and P99 latency by endpoint
histogram_quantile(0.50, sum by(endpoint, le) (rate(http_request_duration_seconds_bucket[5m])))
histogram_quantile(0.99, sum by(endpoint, le) (rate(http_request_duration_seconds_bucket[5m])))

# CPU usage by namespace in Kubernetes
sum by(namespace) (
  rate(container_cpu_usage_seconds_total[5m])
)

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.

Stage 6 · Operate

PromQL in Depth