Trace-Derived Metrics

Why Derive Metrics from Traces?

Trace-derived metrics give you RED metrics without instrumenting each service individually. The OTel Collector or Tempo's metrics generator can produce request rate, error rate, and latency metrics from span streams. This is especially valuable for services you cannot easily instrument.

RED Metrics from Traces

Every span contains the data needed to compute RED metrics. Request rate comes from span count. Error rate comes from span status. Latency comes from span duration. Aggregate these across spans to produce per-service and per-endpoint metrics.

# Request rate from span count
sum(rate(traces_spanmetrics_calls_total[5m]))

# Error rate from span status
sum(rate(traces_spanmetrics_calls_total{status_code="STATUS_CODE_ERROR"}[5m]))
  / sum(rate(traces_spanmetrics_calls_total[5m]))

# Latency P99 from span duration
histogram_quantile(0.99,
  sum by(le) (rate(traces_spanmetrics_duration_milliseconds_bucket[5m]))
)

Service Graph Metrics

Service graph metrics describe the relationships between services. They track request count, error count, and latency between every pair of communicating services. These metrics power dependency graphs and service mesh visualizations.

# Request rate between services
sum(rate(traces_service_graph_request_total[5m])) by (source, target)

# Error rate between services
sum(rate(traces_service_graph_request_failed_total[5m])) by (source, target)

# Latency between services
histogram_quantile(0.99,
  sum by(le, source, target) (rate(traces_service_graph_request_duration_milliseconds_bucket[5m]))
)

Latency Histograms

Trace-derived latency histograms provide the same data as instrumented histograms but from span duration. This is useful for services that do not expose Prometheus metrics but do emit traces.

processors:
  spanmetrics:
    histogram:
      explicit:
        buckets:
          - 5ms
          - 10ms
          - 25ms
          - 50ms
          - 100ms
          - 250ms
          - 500ms
          - 1s
          - 2.5s
          - 5s
          - 10s
    dimensions:
      - name: http.method
      - name: http.status_code

Tempo Metrics Generator

Tempo's metrics generator automatically produces RED metrics and service graph metrics from incoming traces. It runs as a Tempo component and writes metrics to Prometheus via remote_write. No additional Collector configuration is needed.

metrics_generator:
  ring:
    kvstore:
      store: inmemory
  processor:
    service_graphs:
      dimensions:
        - http.method
    span_metrics:
      dimensions:
        - http.method
        - http.status_code
  storage:
    path: /var/tempo/generator/wal
    remote_write:
      - url: http://prometheus:9090/api/v1/write
        send_exemplars: true

Exemplars from Traces

Trace-derived metrics automatically include exemplars. Each metric data point links to a specific trace. This enables the metrics-to-traces correlation without any additional instrumentation.