Stage 5 · Platform
Production Troubleshooting & Operations
Change-Safe Operations
PDBs, drain budgets, maintenance windows, preflight checks, and rollback runbooks.
PDB Operations
PodDisruptionBudgets (PDBs) protect against voluntary disruptions: node drains, cluster upgrades, and scaling events. They ensure a minimum number of pods are always running. PDBs don't protect against involuntary disruptions (OOMKilled, node failure).
# Minimum available pods
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: api-pdb
namespace: production
spec:
minAvailable: 2
selector:
matchLabels:
app: api
---
# Maximum unavailable pods
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: web-pdb
namespace: production
spec:
maxUnavailable: 1
selector:
matchLabels:
app: web
---
# Percentage-based PDB
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: worker-pdb
namespace: production
spec:
minAvailable: "50%"
selector:
matchLabels:
app: workerminAvailable: 2 ensures at least 2 pods are always running. maxUnavailable: 1 allows at most 1 pod to be unavailable. minAvailable: 50% ensures at least half the pods are running.
Drain Budgets
Drain budgets control how many pods can be drained simultaneously during node maintenance. Use --max-per-node to limit concurrent drains. Combine with PDBs for granular control.
# Drain single node
kubectl drain worker-1 \
--ignore-daemonsets \
--delete-emptydir-data \
--grace-period=60 \
--timeout=300s
# Drain multiple nodes sequentially
for node in worker-1 worker-2 worker-3; do
echo "Draining $node"
kubectl cordon $node
kubectl drain $node --ignore-daemonsets --delete-emptydir-data --timeout=300s
echo "Maintenance on $node"
# Perform maintenance...
kubectl uncordon $node
echo "$node recovered"
doneDrain respects PDBs — it waits for pods to be rescheduled before terminating them. --timeout prevents hanging if PDBs prevent eviction. Always cordon before draining to prevent new pods from being scheduled.
Maintenance Windows
Define maintenance windows for cluster operations: upgrades, node rotations, and disruptions. Use tools like Kured for automated reboots within windows. Communicate windows to all stakeholders.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kured
namespace: kube-system
spec:
template:
spec:
containers:
- name: kured
image: ghcr.io/kubereboot/kured:1.14.0
args:
- --schedule=0 2 * * * # 2 AM daily
- --reboot-sentinel=/var/run/reboot-required
- --period=1h
- --ds-name=kured
- --ds-namespace=kube-system
- --drain-grace-period=60
- --drain-timeout=300
- --skip-wait-for-delete-timeout=60
- --concurrency=1 # One node at a time
- --start-time=02:00
- --end-time=06:00
- --time-zone=UTCKured runs on every node and checks for /var/run/reboot-required. During the maintenance window (2 AM - 6 AM UTC), it drains the node, reboots it, and uncordons it. concurrency: 1 ensures only one node reboots at a time.
Preflight Checks
Preflight checks verify the cluster is healthy before operations. Check: PDBs allow disruptions, sufficient capacity, no active incidents, and backups are current. Automate these checks in runbooks.
#!/bin/bash
echo "=== Preflight Checks ==="
# Check PDBs
echo "Checking PDBs..."
kubectl get pdb --all-namespaces -o json | jq '.items[] |
select(.status.disruptionsAllowed == 0) |
{name: .metadata.name, namespace: .metadata.namespace}'
# Check node health
echo "Checking node health..."
kubectl get nodes | grep -v "Ready"
# Check pending pods
echo "Checking pending pods..."
kubectl get pods --field-selector status.phase=Pending --all-namespaces
# Check recent events
echo "Checking warning events..."
kubectl get events --all-namespaces --field-selector type=Warning --sort-by='.lastTimestamp' | tail -20
# Check backup status
echo "Checking Velero backups..."
velero backup get | grep -v "Completed"
echo "=== Preflight Complete ==="Preflight checks verify: PDBs allow disruptions (disruptionsAllowed > 0), nodes are Ready, no pods are Pending, no warning events, and backups are current. Fix any issues before proceeding.
Rollback Runbooks
Rollback runbooks provide step-by-step procedures for reverting changes. Document: what to roll back, how to roll back, how to verify, and who to notify. Practice rollbacks regularly.
# Rollback Runbook: API Deployment
## Trigger
- Error rate > 5% for 5 minutes
- P99 latency > 2s for 10 minutes
- Critical bug reported
## Steps
1. Identify current version
kubectl get deployment api -o jsonpath='{.spec.template.spec.containers[0].image}'
2. Rollback to previous version
kubectl rollout undo deployment/api -n production
3. Or rollback to specific revision
kubectl rollout history deployment/api -n production
kubectl rollout undo deployment/api --to-revision=3 -n production
4. Verify rollback
kubectl rollout status deployment/api -n production
kubectl get pods -l app=api -n production
5. Validate health
kubectl exec -it test-pod -- curl http://api-service:8080/healthz
6. Notify team
- Post in #incidents channel
- Update status page
- Document in incident logRollback runbooks provide clear, actionable steps. Practice them regularly to ensure they work. Update runbooks after incidents to incorporate lessons learned.
Change Management
- Document all changes in a change log
- Run preflight checks before changes
- Deploy during maintenance windows
- Use PDBs to protect availability
- Monitor health after changes
- Have rollback procedures ready
- Communicate changes to stakeholders
- Post-change review for improvements
Before any production change: 1) PDBs configured, 2) Preflight checks pass, 3) Rollback procedure documented, 4) Monitoring dashboards open, 5) Team notified, 6) Maintenance window active. Never skip these steps.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.