Stage 5 · Platform
Progressive Delivery Controllers
Progressive Delivery Runbooks
Documenting pause, promote, abort, retry, and manual override steps for release incidents.
Why Runbooks?
Runbooks document the exact steps for common operations and incidents. When a canary is failing at 3 AM, the on-call engineer should not be figuring out how to abort — they should follow a documented procedure. Runbooks reduce incident response time and prevent mistakes.
Canary Pause
# Argo Rollouts - pause at current step
kubectl argo rollouts pause myapp -n production
# Flagger - pause canary promotion
kubectl annotate canary/myapp flagger.app/pause=true -n production
# Verify pause status
kubectl argo rollouts status myapp -n production
kubectl get canary myapp -n production -o yamlPausing a canary stops traffic progression at the current step. This is useful when you need more time to evaluate metrics or investigate an issue. The canary continues to receive its current traffic weight.
Canary Promote
# Argo Rollouts - promote to next step
kubectl argo rollouts promote myapp -n production
# Argo Rollouts - promote to full (skip remaining steps)
kubectl argo rollouts promote --full myapp -n production
# Flagger - manually trigger promotion
kubectl annotate canary/myapp flagger.app/promote=true -n production
# Verify promotion
kubectl argo rollouts get rollout myapp -n productionPromoting advances the canary to the next traffic weight step. promote --full skips all remaining steps and goes to 100%. Use promote when you are confident the canary is stable but want to skip remaining analysis steps.
Canary Abort
# Argo Rollouts - abort current rollout
kubectl argo rollouts abort myapp -n production
# Flagger - abort canary
kubectl annotate canary/myapp flagger.app/abort=true -n production
# Kubernetes - rollback deployment
kubectl rollout undo deployment/myapp -n production
# Verify rollback
kubectl argo rollouts status myapp -n production
kubectl get pods -n production -l app=myapp
# Check rollback history
kubectl argo rollouts history myapp -n productionAborting stops the canary and routes all traffic back to the stable version. For Argo Rollouts, this reverts the Rollout to the previous revision. For plain Deployments, kubectl rollout undo reverts to the last successful revision.
Manual Override
# Set explicit traffic weight (bypass analysis)
kubectl argo rollouts set weight myapp 0 -n production
# Force promotion bypassing analysis
kubectl argo rollouts promote --full myapp -n production
# Retry a failed rollout
kubectl argo rollouts retry myapp -n production
# Restart the rollout
kubectl argo rollouts restart myapp -n productionManual overrides bypass the automated analysis. Use them when the automated system is too conservative or when you have additional context. Setting weight to 0 effectively rolls back to stable.
Incident Response
# Release Incident Runbook
## 1. Detect
- [ ] Alert received (Prometheus, PagerDuty, Slack)
- [ ] Identify affected service and version
- [ ] Check canary status: kubectl argo rollouts status <service>
## 2. Assess
- [ ] Check error rate: curl prometheus:9090/api/v1/query?query=...
- [ ] Check latency: p50, p95, p99
- [ ] Check SLO burn rate
- [ ] Check user reports (support tickets, social media)
## 3. Decide
- [ ] If transient (< 2 min, no user impact): monitor
- [ ] If persistent but low impact: pause and investigate
- [ ] If high impact: abort immediately
## 4. Act
- [ ] Abort: kubectl argo rollouts abort <service> -n production
- [ ] Notify: post in #incidents channel
- [ ] Verify: confirm traffic shifted to stable
## 5. Investigate
- [ ] Collect logs: kubectl logs -l app=<service> --tail=100
- [ ] Collect metrics: check dashboards for anomalies
- [ ] Collect traces: check Jaeger/Zipkin for errors
## 6. Recover
- [ ] Fix root cause
- [ ] Deploy fix through normal pipeline
- [ ] Verify fix in staging before production
## 7. Review
- [ ] Write post-mortem
- [ ] Update runbook if needed
- [ ] Create follow-up action itemsThis checklist provides a structured approach to release incidents. Each step has specific actions and verification points. Adapt this template to your organization's specific tools and processes.
Store runbooks in the same repository as your application code. This keeps them version-controlled, reviewable, and always up-to-date. Link runbooks from your monitoring dashboards for quick access during incidents.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.