Stage 5 · Platform
Progressive Delivery Controllers
Rollback Automation
Reverting failed releases from SLO burn, synthetic probes, Kubernetes conditions, and error rates.
Rollback Triggers
Automated rollback triggers detect degradation and revert to a known-good state. The key is choosing triggers that are accurate (low false positives) and fast (quick detection). Common triggers include SLO violations, synthetic test failures, error rate spikes, and Kubernetes resource conditions.
SLO-Based Rollback
groups:
- name: slo-burn-rate
rules:
- alert: HighSLOBurnRate
expr: |
(
1 - (
sum(rate(http_requests_total{status!~"5.."}[1h]))
/ sum(rate(http_requests_total[1h]))
)
) > (14.4 * 0.001)
and
(
1 - (
sum(rate(http_requests_total{status!~"5.."}[5m]))
/ sum(rate(http_requests_total[5m]))
)
) > (14.4 * 0.001)
for: 2m
labels:
severity: critical
action: rollback
annotations:
summary: "SLO burn rate exceeds 14.4x"
description: "At current rate, 30-day error budget exhausts in 2 days"
- alert: LowSLOBurnRate
expr: |
(
1 - (
sum(rate(http_requests_total{status!~"5.."}[6h]))
/ sum(rate(http_requests_total[6h]))
)
) > (6 * 0.001)
and
(
1 - (
sum(rate(http_requests_total{status!~"5.."}[30m]))
/ sum(rate(http_requests_total[30m]))
)
) > (6 * 0.001)
for: 15m
labels:
severity: warning
action: investigate
annotations:
summary: "SLO burn rate exceeds 6x"SLO burn rate alerts use two time windows (1h/5m and 6h/30m) to reduce false positives. The 14.4x burn rate is a critical threshold that exhausts the monthly budget in 2 days. The action label triggers automated rollback.
Synthetic Probe Rollback
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: synthetic-probes
spec:
metrics:
- name: health-check
interval: 30s
count: 10
successCondition: result[0] == 200
failureLimit: 3
provider:
webhook:
url: http://synthetic-monitor:8080/probe
timeout: 10s
method: GET
headers:
- key: X-Probe-Target
value: "myapp-canary.production:8080"
- name: e2e-smoke
interval: 2m
count: 3
successCondition: result[0] == "pass"
failureLimit: 1
provider:
webhook:
url: http://synthetic-monitor:8080/e2e/smoke
timeout: 60s
method: POST
body: |
{
"service": "myapp",
"environment": "production",
"tests": ["login", "checkout", "health"]
}Synthetic probes run automated tests against the canary. The health-check verifies basic availability. The e2e-smoke test verifies critical user flows. If either fails, the rollout is aborted.
Kubernetes Conditions
apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-cm
namespace: argocd
data:
resource.customizations.health.apps_Deployment: |
hs = {}
hs.status = "Progressing"
hs.message = "Waiting for rollout"
if obj.status ~= nil then
-- Check for available replicas
if obj.status.availableReplicas ~= nil and
obj.status.availableReplicas == obj.spec.replicas then
hs.status = "Healthy"
hs.message = "All replicas available"
end
-- Check for unavailable replicas
if obj.status.unavailableReplicas ~= nil and
obj.status.unavailableReplicas > 0 then
hs.status = "Degraded"
hs.message = obj.status.unavailableReplicas .. " replicas unavailable"
end
-- Check for failed pods
if obj.status.conditions ~= nil then
for _, condition in ipairs(obj.status.conditions) do
if condition.type == "Progressing" and
condition.status == "False" then
hs.status = "Degraded"
hs.message = "Rollout stalled: " .. (condition.reason or "unknown")
end
if condition.type == "ReplicaFailure" and
condition.status == "True" then
hs.status = "Degraded"
hs.message = "Replica failure: " .. (condition.message or "unknown")
end
end
end
end
return hsKubernetes deployment conditions provide structured status information. ArgoCD health checks use these conditions to determine deployment health. Degraded conditions trigger rollback automatically.
Error Rate Rollback
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: error-rate-check
spec:
args:
- name: service-name
- name: namespace
metrics:
- name: http-error-rate
interval: 30s
count: 10
successCondition: result[0] <= 0.5
failureLimit: 3
provider:
prometheus:
address: http://prometheus.monitoring:9090
query: |
(
sum(rate(http_requests_total{
service="{{args.service-name}}",
namespace="{{args.namespace}}",
status=~"5.."
}[1m])) /
sum(rate(http_requests_total{
service="{{args.service-name}}",
namespace="{{args.namespace}}"
}[1m]))
) * 100
- name: grpc-error-rate
interval: 30s
count: 10
successCondition: result[0] <= 1
failureLimit: 3
provider:
prometheus:
address: http://prometheus.monitoring:9090
query: |
(
sum(rate(grpc_server_handled_total{
service="{{args.service-name}}",
namespace="{{args.namespace}}",
grpc_code!="OK"
}[1m])) /
sum(rate(grpc_server_handled_total{
service="{{args.service-name}}",
namespace="{{args.namespace}}"
}[1m]))
) * 100Error rate analysis checks HTTP and gRPC error rates. successCondition: result[0] <= 0.5 means the error rate must be below 0.5%. failureLimit: 3 allows 3 consecutive failures before aborting.
Rollback Orchestration
- Detect — identify degradation through metrics, probes, or conditions.
- Decide — evaluate whether the issue is transient or persistent.
- Revert — shift traffic back to the stable version.
- Notify — alert the team about the rollback and its cause.
- Investigate — collect logs, metrics, and traces for post-mortem.
Automated rollback is a feature, not a failure. It means your system can recover without human intervention. Teams that embrace automated rollback ship faster because they have confidence that bad changes are quickly reverted.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.