Cardinality and Cost Control

What Is Cardinality?

Cardinality is the number of unique label combinations for a metric. A metric with 10 jobs and 5 endpoints has 50 unique series. A metric with user_id labels and 1 million users has 1 million series per job. High cardinality is the most common cause of Prometheus performance issues.

Cost Impact

Detecting High Cardinality

Monitor Prometheus's own metrics to detect cardinality growth. The TSDB status endpoint shows the highest cardinality metrics. Regularly review series count per metric to catch issues early.

# Query TSDB status
curl -s http://localhost:9090/api/v1/status/tsdb | jq '.data.totalSeries'

# Find highest cardinality metrics
curl -s http://localhost:9090/api/v1/status/tsdb | jq '.data.highestCardinalityMetricNames[:10]'

# Count series per metric
curl -s 'http://localhost:9090/api/v1/query?query=count by (__name__)({__name__=~".+"})' | jq '.data.result[:10]'

# Total series count
prometheus_tsdb_head_series

# Series per metric
topk(10, count by(__name__)({__name__=~".+"}))

Mitigation Strategies

• Use relabel_configs to drop high-cardinality labels before storage.
• Aggregate high-cardinality metrics with recording rules.
• Use hashmod relabeling to sample metrics across Prometheus instances.
• Replace user_id, request_id, and trace_id with aggregation.
• Use bucket labels instead of exact values for numeric data.
• Review metric naming to avoid per-request labels.

Drop Rules

metric_relabel_configs:
  # Drop high-cardinality labels
  - source_labels: [user_id]
    action: labeldrop

  # Drop entire metrics
  - source_labels: [__name__]
    regex: "debug_.*"
    action: drop

  # Drop specific label combinations
  - source_labels: [path]
    regex: "/api/v[0-9]+/users/[0-9]+"
    action: drop

Metric Naming Reviews

Regularly review metric naming during code reviews. Flag metrics that include user IDs, request IDs, or other high-cardinality values as labels. Establish team guidelines for acceptable label cardinality.

# High cardinality (bad)
http_request_duration_seconds{path="/users/12345", request_id="abc-123"}

# Low cardinality (good)
http_request_duration_seconds{method="GET", endpoint="/users/:id", status="200"}