Stage 5 · Platform
Compute & Containers
AKS Upgrades & Operations
Blue-green node pool upgrades, node drain, and zero-downtime cluster maintenance.
AKS Upgrade Model
AKS upgrades are applied in two phases: the control plane (API server, etcd) upgrades first, then the node pools upgrade. The control plane always runs a version one minor version newer than or equal to the oldest node pool.
# List available control plane upgrades
az aks get-upgrades \
--resource-group rg-aks-prod \
--name aks-payments-prod \
--query "controlPlaneProfile.upgrades[].{version:kubernetesVersion, isUpgradable:isUpgradable}" \
--output table
# List available node pool upgrades
az aks nodepool get-upgrades \
--resource-group rg-aks-prod \
--cluster-name aks-payments-prod \
--nodepool-name pool-app \
--query "upgrades[].{version:kubernetesVersion}" \
--output tableAKS supports upgrading one minor version at a time (e.g., 1.28 to 1.29). You cannot skip versions. Plan for upgrades every 3 months as Kubernetes releases follow a quarterly cadence.
Control Plane Upgrades
Control plane upgrades are fully managed by Azure. The API server briefly restarts during the upgrade — typically under 60 seconds. Existing pods continue running, but you cannot deploy new resources during the restart window.
# Upgrade the control plane to a specific version
az aks upgrade \
--resource-group rg-aks-prod \
--name aks-payments-prod \
--kubernetes-version 1.29.4 \
--control-plane-only
# Check current cluster version
az aks show \
--resource-group rg-aks-prod \
--name aks-payments-prod \
--query "{version:kubernetesVersion, autoUpgradeChannel:autoUpgradeChannel}" \
--output jsonUse --control-plane-only to upgrade just the control plane without touching node pools. This is useful when you want to test the control plane before upgrading nodes.
Use az aks update --auto-upgrade-channel stable to let Azure automatically upgrade your control plane and node pools. This keeps clusters patched but is only recommended for dev/test.
Node Pool Upgrades
Node pool upgrades replace nodes one at a time. The upgrade orchestrates cordoning, draining, and replacing each node. Pods are evicted and rescheduled on other nodes during the process.
# Upgrade a node pool to a new Kubernetes version
az aks nodepool upgrade \
--resource-group rg-aks-prod \
--cluster-name aks-payments-prod \
--name pool-app \
--kubernetes-version 1.29.4
# Upgrade all node pools at once
az aks upgrade \
--resource-group rg-aks-prod \
--name aks-payments-prod \
--kubernetes-version 1.29.4Upgrading all node pools at once is faster but risks longer downtime if issues arise. Upgrade node pools individually for safer rollouts.
Blue-Green Node Pool Upgrades
Blue-green upgrades create a new node pool alongside the old one, migrate workloads, and delete the old pool. This provides a rollback path and zero-downtime upgrades.
# 1. Create a new node pool with the target version
az aks nodepool add \
--resource-group rg-aks-prod \
--cluster-name aks-payments-prod \
--name pool-app-v2 \
--node-count 3 \
--node-vm-size Standard_D4s_v3 \
--mode User \
--labels workload=app
# 2. Cordon the old pool (prevent new pods)
kubectl cordon <old-pool-node-1>
kubectl cordon <old-pool-node-2>
# 3. Drain old nodes (evict all pods)
kubectl drain <old-pool-node-1> --ignore-daemonsets --delete-emptydir-data
kubectl drain <old-pool-node-2> --ignore-daemonsets --delete-emptydir-data
# 4. Verify workloads are running on new pool
kubectl get pods -o wide
# 5. Delete the old node pool
az aks nodepool delete \
--resource-group rg-aks-prod \
--cluster-name aks-payments-prod \
--name pool-appBlue-green upgrades give you a rollback path — if the new pool has issues, re-cordon the new nodes and drain them back. The old pool stays until you explicitly delete it.
Node Drain and PDBs
kubectl drain removes all pods from a node. Pod Disruption Budgets (PDBs) protect against too many pods being evicted simultaneously — the drain will block if a PDB would be violated.
# Create a PDB that allows at most 1 pod to be unavailable
cat <<EOF | kubectl apply -f -
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: payments-pdb
namespace: default
spec:
maxUnavailable: 1
selector:
matchLabels:
app: payments
EOF
# Verify the PDB
kubectl get pdb -n defaultPDBs are critical for production workloads. Without a PDB, a node drain could evict all replicas simultaneously, causing downtime.
Without PDBs, node upgrades can cause cascading failures. Every production deployment should have a PDB to ensure at least one replica remains running during maintenance.
Ongoing Cluster Maintenance
AKS clusters require regular maintenance beyond version upgrades. These tasks keep clusters healthy and secure.
- Kubernetes version upgrades — Plan for quarterly upgrades within 3 months of a new version release
- Node OS security patching — Enable --node-image auto-upgrade for automatic OS patches
- Certificate rotation — AKS auto-rotates certificates; verify with az aks show
- Cluster autoscaler tuning — Review --min/--max counts based on actual usage patterns
- Resource quota review — Update resource quotas as workloads grow
- Network policy audit — Review and update policies as new services are deployed
#!/usr/bin/env bash
set -euo pipefail
CLUSTER="aks-payments-prod"
RG="rg-aks-prod"
# Check node image upgrade status
az aks nodepool list \
--resource-group "$RG" \
--cluster-name "$CLUSTER" \
--query "[].{name:name, imageVersion:currentNodeImageVersion}" --output table
# Trigger node image upgrade
az aks nodepool upgrade \
--resource-group "$RG" \
--cluster-name "$CLUSTER" \
--name pool-app \
--node-image-onlyRun node image upgrades weekly to keep nodes patched. The --node-image-only flag upgrades only the OS image, not the Kubernetes version.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.