Correlation and Exemplars

Why Correlation Matters

Metrics tell you something is wrong. Logs tell you what happened. Traces tell you where the problem occurred. Correlation links these three pillars so you can move seamlessly between them during an investigation.

• Metrics show error rate is elevated.
• Exemplar links to a specific trace showing the error.
• Trace shows which downstream service caused the failure.
• Logs from that service reveal the exact error message.

Exemplars

Exemplars attach trace IDs to metric data points. When you see an unusual spike in a metric, the exemplar links to a specific trace that contributed to that value. This bridges metrics and traces without manual correlation.

# Exemplars are attached to histogram metrics
# Click on a data point in Grafana to see the linked trace

# Exemplar query (Grafana handles this automatically)
histogram_quantile(0.99,
  sum by(le) (rate(http_request_duration_seconds_bucket[5m]))
)

Trace to Logs

From a trace in Grafana Tempo, you can jump directly to logs for the same trace. This requires including the trace_id in your log entries. The Grafana UI creates a Loki query filtered to that trace ID.

{service="api"} | json | trace_id="abc123def456789"

Logs to Traces

From a log line in Loki, you can jump to the associated trace in Tempo. This requires the trace_id field in your structured logs. Grafana renders a clickable link to Tempo Explore for that trace ID.

# In Grafana Loki datasource config
datasources:
  - name: Loki
    type: loki
    jsonData:
      derivedFields:
        - datasourceUid: Tempo
          matcherRegex: '"trace_id":"(\w+)"'
          name: TraceID
          url: "$${__value.raw}"
          matcherType: regex

Grafana Explore

Grafana Explore is the investigation workspace. It provides split-pane views for querying Prometheus, Loki, and Tempo side by side. Use Explore to pivot between metrics, logs, and traces during incident response.

Implementation

• Include trace_id in every structured log entry.
• Add exemplars to histogram metrics with trace IDs.
• Configure derived fields in Loki datasource for trace links.
• Configure tracesToLogs in Tempo datasource for log links.
• Use OpenTelemetry to propagate trace context across services.
• Test correlation by following a request from metrics to traces to logs.