Stage 5 · Platform
Platform Engineering & Multi-Tenancy
Day 2 Operations
Upgrades, backup (Velero), disaster recovery, and cluster lifecycle.
Cluster Upgrades
Kubernetes follows a quarterly release cycle. Each release is supported for roughly 14 months. Upgrades must be done one minor version at a time (e.g., 1.28 to 1.29, skipping 1.30 is not supported). Always read the release notes and upgrade guide before upgrading.
# Check available upgrades
az aks get-upgrades --resource-group my-rg --name my-cluster
# Upgrade control plane and nodes
az aks upgrade \
--resource-group my-rg \
--name my-cluster \
--kubernetes-version 1.29.0 \
--control-plane-only # Upgrade control plane first
# Upgrade node pools separately
az aks nodepool upgrade \
--resource-group my-rg \
--cluster-name my-cluster \
--name systempool \
--kubernetes-version 1.29.0On AKS, control plane and node pools upgrade separately. Upgrade the control plane first, then node pools one at a time. Node pools can run up to 2 minor versions behind the control plane.
Node Upgrades
Node upgrades drain pods, cordon the node, upgrade the kubelet, and uncordon. Use PDBs to prevent downtime during node upgrades. Ensure your applications have sufficient replica counts and pod disruption budgets.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: web-pdb
namespace: production
spec:
minAvailable: 2 # At least 2 pods must always be running
selector:
matchLabels:
app: web
---
# Or use maxUnavailable
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: api-pdb
namespace: production
spec:
maxUnavailable: 1 # At most 1 pod can be unavailable
selector:
matchLabels:
app: apiPDBs prevent voluntary disruptions (drain, eviction) from reducing availability below the threshold. During node upgrades, Kubernetes respects PDBs — it waits for pods to be rescheduled before draining the node.
Backup with Velero
Velero backs up Kubernetes resources and persistent volumes. It creates tar archives of cluster state and snapshots of PVs. Velero supports scheduled backups, incremental backups, and cross-cluster migration.
apiVersion: velero.io/v1
kind: Schedule
metadata:
name: daily-backup
namespace: velero
spec:
schedule: "0 2 * * *" # 2 AM daily
template:
includedNamespaces:
- production
- staging
excludedResources:
- events
- events.events.k8s.io
storageLocation: default
volumeSnapshotLocations:
- default
ttl: 720h # 30 days retention
includedClusterResources: true
---
apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
name: default
namespace: velero
spec:
provider: azure
objectStorage:
bucket: velero-backups
config:
resourceGroup: my-backup-rg
storageAccount: veleroacctVelero Schedule creates backups automatically. includedNamespaces specifies which namespaces to back up. volumeSnapshotLocations define where PV snapshots are stored. ttl controls retention period.
Velero backs up cluster state but cannot restore the cluster itself. If the control plane is lost, you need a new cluster first. Velero restores resources and data to the new cluster. Always test restore procedures.
Disaster Recovery
Disaster recovery requires: regular etcd backups, Velero backups for resources and PVs, infrastructure as code for cluster provisioning, and documented runbooks for restoration. Practice DR drills quarterly.
- etcd backup — Snapshots every hour, retained for 7 days
- Velero backup — Daily full backups, hourly incremental
- Infrastructure as code — Terraform/Pulumi for cluster provisioning
- DNS failover — Pre-configured DNS entries for rapid cutover
- Runbooks — Step-by-step restoration procedures
- DR drills — Quarterly practice to verify procedures
Cluster Lifecycle Management
Cluster lifecycle management covers provisioning, upgrades, scaling, and decommissioning. Use tools like Cluster API (CAK), Terraform, or cloud-managed services (EKS, AKS, GKE) to automate cluster lifecycle operations.
# List clusters
kubectl get clusters --all-namespaces
# Scale a node pool
kubectl scale cluster/my-cluster --replicas=5
# Upgrade cluster version
kubectl patch cluster/my-cluster --type merge -p \
'{"spec":{"version":"1.29.0"}}'
# Check cluster conditions
kubectl get cluster my-cluster -o yaml | jq .status.conditionsCluster API manages clusters declaratively. You define the desired state (version, node count, machine type) and the controllers reconcile the actual state. This works across cloud providers with provider-specific controllers.
Maintenance Windows
Define maintenance windows for cluster operations: upgrades, node rotations, and disruptions. Use node maintenance annotations and PDBs to control when and how maintenance occurs.
# Annotate nodes during maintenance
kubectl annotate node worker-1 \
node.alpha.kubernetes.io/maintenance-window="Sat 02:00-06:00 UTC"
# Cordon node before maintenance
kubectl cordon worker-1
# Drain with PDB respect
kubectl drain worker-1 \
--ignore-daemonsets \
--delete-emptydir-data \
--grace-period=60 \
--timeout=300s
# Perform maintenance...
# Uncordon after maintenance
kubectl uncordon worker-1cordon marks the node as unschedulable. drain evicts pods respecting PDBs. --ignore-daemonsets keeps system pods running. --delete-emptydir-data removes pods using emptyDir. --grace-period gives pods time to shut down gracefully.
Kured (Kubernetes Reboot Daemon) automates node reboots after kernel updates. It drains nodes, reboots them, and uncordons them. Configure reboot windows to limit when reboots can occur.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.