Delivery Observability

DORA metrics, deployment traces, reconciliation alerts, and rollback audit trails — measuring and monitoring delivery performance.

6 min readInfrastructure as CodeProvision

DORA Metrics

DORA metrics measure delivery performance. They are the gold standard for measuring engineering team effectiveness. Four metrics matter: deployment frequency, lead time, change failure rate, and time to restore.

Metric	What It Measures	Elite Target
Deployment Frequency	How often you deploy	On-demand (multiple per day)
Lead Time for Changes	Time from commit to production	Less than one hour
Change Failure Rate	Percentage of deployments causing failure	0-15%
Time to Restore Service	Time to recover from failure	Less than one hour

Measuring deployment frequency


# Prometheus metric for deployment count
- record: gitops:deployments:total
  expr: |
    sum by (namespace) (
      increase(argocd_app_sync_total{phase="Succeeded"}[24h])
    )

# Prometheus metric for lead time
- record: gitops:lead_time:seconds
  expr: |
    argocd_app_reconcile_seconds_sum{phase="Succeeded"}
    /
    argocd_app_reconcile_seconds_count{phase="Succeeded"}

These recording rules compute DORA metrics from ArgoCD data. Deployment frequency counts successful syncs per day. Lead time is derived from reconciliation duration.

Deployment Traces

Deployment event recording


apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
data:
  trigger.on-sync-succeeded: |
    - when: app.status.operationState.phase in ['Succeeded']
      send: [deployment-success]
  template.deployment-success: |
    webhook:
      method: POST
      body: |
        {
          "app": "{{.app.metadata.name}}",
          "revision": "{{.app.status.sync.revision}}",
          "timestamp": "{{.app.status.operationState.finishedAt}}",
          "duration": "{{.app.status.operationState.operation.sync.revision}}"
        }

The notification template records deployment events with the app name, revision, timestamp, and duration. This data feeds into deployment trace dashboards and audit systems.

Reconciliation Alerts

Prometheus alerting rules


apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: gitops-reconciliation-alerts
spec:
  groups:
    - name: gitops
      rules:
        - alert: ReconciliationStalled
          expr: flux_reconcile_duration_seconds{result="error"} > 300
          for: 10m
          labels:
            severity: critical
          annotations:
            summary: "Flux reconciliation stalled for {{ $labels.name }}"

        - alert: SyncDriftDetected
          expr: argocd_app_info{status="OutOfSync"} == 1
          for: 30m
          labels:
            severity: warning
          annotations:
            summary: "Application {{ $labels.name }} is out of sync for 30+ minutes"

        - alert: DeploymentFailed
          expr: argocd_app_sync_total{phase="Error"} > 0
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "Deployment failing for {{ $labels.name }}"

These alerts fire when reconciliation is stalled, drift is detected, or deployments are failing. The for duration prevents alerting on transient issues. Critical alerts indicate broken deployments that need immediate attention.

Rollback Audit Trails

ArgoCD rollback tracking


# ArgoCD tracks rollback history
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
status:
  history:
    - revision: abc123
      deployedAt: "2024-01-15T10:00:00Z"
      initiatedBy:
        username: admin
      source:
        path: apps/my-app
    - revision: def456
      deployedAt: "2024-01-15T09:00:00Z"
      initiatedBy:
        username: automated
      source:
        path: apps/my-app
      operationState:
        phase: RolledBack

ArgoCD history tracks every deployment and rollback. The initiatedBy field shows who triggered the change. The operationState shows rollback status. This provides a complete audit trail.

Observability Dashboards

Sync status dashboard — overview of all application sync states.
Deployment frequency chart — deployments per day, per team, per environment.
Lead time distribution — time from commit to production deployment.
Reconciliation performance — success rate, duration, and error counts.
Drift detection timeline — when drift was detected and how it was resolved.

Continuous Improvement

Use DORA metrics to identify bottlenecks. If lead time is high, focus on reducing testing time. If change failure rate is high, improve policy checks and testing. If time to restore is high, improve rollback automation.

Track metrics over time

DORA metrics are most valuable when tracked over time. A snapshot is less useful than a trend. Set up dashboards that show weekly or monthly trends. Celebrate improvements and investigate regressions.

GitOps makes observability automatic

GitOps provides built-in observability — Git history, sync status, and reconciliation logs. Most of the observability infrastructure is already in place. You just need to surface it in dashboards and alerts.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.

Stage 4 · Provision

Continuous Delivery & GitOps