Annotations and Events

Annotations Overview

Annotations are vertical markers on Grafana panels that show when events occurred. They help correlate metric changes with deployments, incidents, configuration changes, or any external event. Annotations turn dashboards from metric displays into investigation tools.

Annotation Sources

Prometheus-Based Annotations

Grafana can automatically create annotations when Prometheus alerts fire or resolve. Enable this by linking an Alertmanager datasource and selecting which alert rules should create annotations.

# In Grafana datasource config
apiVersion: 1

datasources:
  - name: Alertmanager
    type: alertmanager
    url: http://alertmanager:9093
    jsonData:
      implementation: prometheus

Kubernetes Events

Kubernetes events show pod scheduling, scaling, restarts, and evictions. Overlaying these on Grafana dashboards reveals infrastructure-level changes that affect service metrics. Use kube-state-metrics or the events API to surface these events.

# Pod restart count increase
increase(kube_pod_container_status_restarts_total[1h]) > 0

# Pod not ready
kube_pod_status_ready{condition="false"} == 1

Custom Annotations API

Push custom annotations using the Grafana API. This lets CI/CD pipelines mark deployments, feature flag services mark toggles, or any system mark important events directly on dashboards.

curl -X POST http://grafana:3000/api/annotations \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $API_KEY" \
  -d '{
    "dashboardUID": "my-dashboard",
    "time": 1700000000000,
    "text": "Deployed v2.3.1 to production",
    "tags": ["deploy", "production"]
  }'

Best Practices

• Automate annotations for all production deployments.
• Include the version, author, and change summary in annotation text.
• Use annotations to mark incident start and end times.
• Do not annotate every small change. Focus on events that affect user-facing behavior.
• Keep annotation text concise but informative for future reference.