Canary Releases

Shifting traffic with service meshes, ingress controllers, metrics checks, and weighted routing.

7 min readCI/CD & GitOpsPlatform

Canary Release Concept

A canary release deploys the new version to a small subset of users before rolling it out to everyone. The name comes from coal mining — miners used canaries to detect toxic gases. Similarly, a canary deployment detects problems before they affect all users.

Traffic is gradually shifted from the old version to the new version. If the canary shows errors, high latency, or degraded metrics, traffic is shifted back. This limits the blast radius of bad deployments.

NGINX Ingress Canary

NGINX canary annotations


# Stable deployment
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp-stable
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
    - host: myapp.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: myapp-stable
                port:
                  number: 80

---
# Canary deployment
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp-canary
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "10"
spec:
  rules:
    - host: myapp.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: myapp-canary
                port:
                  number: 80

The canary Ingress uses the canary annotation to enable traffic splitting. canary-weight: 10 sends 10% of traffic to the canary. Increase the weight gradually: 10%, 25%, 50%, 100%. Remove the canary Ingress when fully promoted.

Istio Traffic Splitting

Istio VirtualService traffic split


apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: myapp
spec:
  hosts:
    - myapp
  http:
    - route:
        - destination:
            host: myapp
            subset: stable
          weight: 90
        - destination:
            host: myapp
            subset: canary
          weight: 10

---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: myapp
spec:
  host: myapp
  subsets:
    - name: stable
      labels:
        version: v1
    - name: canary
      labels:
        version: v2

Istio VirtualService splits traffic between stable and canary subsets. DestinationRule defines which pods belong to each subset based on labels. The weight values must sum to 100.

Promotion Criteria

Before promoting a canary, verify these metrics are within acceptable thresholds. Define these criteria before the deployment, not during.

Error rate — the canary error rate should not exceed the baseline by more than 0.5%.
Latency — p95 latency should be within 10% of the baseline.
Saturation — CPU and memory usage should be comparable to the baseline.
Business metrics — conversion rates, order completion, or other KPIs should not degrade.

Prometheus canary check


apiVersion: flagger.app/v1beta1
kind: Canary
metadata:
  name: myapp
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  progressDeadlineSeconds: 600
  analysis:
    interval: 1m
    threshold: 5
    maxWeight: 50
    stepWeight: 10
    metrics:
      - name: request-success-rate
        thresholdRange:
          min: 99
        interval: 1m
      - name: request-duration
        thresholdRange:
          max: 500
        interval: 1m

Flagger monitors Prometheus metrics during the canary. If the success rate drops below 99% or latency exceeds 500ms, the canary is automatically rolled back. stepWeight increases traffic by 10% each interval if metrics are healthy.

Automated Canary Analysis

GitLab CI automated canary


deploy-canary:
  stage: deploy
  script:
    - kubectl set image deployment/myapp-canary myapp=${IMAGE}
    - kubectl rollout status deployment/myapp-canary --timeout=300s

  after_script:
    - |
      echo "Waiting 5 minutes for canary metrics..."
      sleep 300
      ERROR_RATE=$(curl -s "http://prometheus:9090/api/v1/query" \
        --data-urlencode 'query=rate(http_requests_total{status=~"5..",version="canary"}[5m]) /
          rate(http_requests_total{version="canary"}[5m]) * 100' \
        | jq -r '.data.result[0].value[1]')
      if (( $(echo "$ERROR_RATE > 1" | bc -l) )); then
        echo "Canary error rate $ERROR_RATE% exceeds threshold"
        kubectl rollout undo deployment/myapp-canary
        exit 1
      fi

The after_script waits for metrics to accumulate, queries Prometheus for the canary error rate, and rolls back if the threshold is exceeded. This is a simple automated canary analysis without external tools.

Canary Rollback

Canary rollback is fast because the stable version is still running at full capacity. To rollback, shift 100% of traffic back to stable. With Kubernetes, this means updating the VirtualService weight or removing the canary Ingress annotation.

Keep the previous version running

Do not scale down the stable deployment during a canary. Keep it running at full capacity so you can rollback instantly. Only scale down stable after the canary is fully promoted and verified.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.

Stage 5 · Platform

Deployment Strategies