Platform Engineering
The Day a `kubectl drain` Took Down Payments (and the PDB We Didn't Have)
A routine node drain during a cluster upgrade evicted every replica of a critical service within seconds of each other, because nothing told Kubernetes it wasn't allowed to.
"3 replicas" felt safe. That was the entire justification for why nobody worried about the payments-api deployment during routine node maintenance. Three replicas, spread across three nodes, should mean losing one node never takes the whole service down. It should. What actually happened during a routine kubectl drain as part of a cluster node upgrade was that all three replicas happened to land on the two nodes being drained back to back, and Kubernetes evicted all three within about eight seconds of each other, because nothing was in place to tell it not to.
What `kubectl drain` actually does (and doesn't check by default)
kubectl drain cordons a node (stops new pods from scheduling there) and then evicts every pod currently running on it, so the node can be safely taken down for maintenance. By default, Kubernetes has no built-in knowledge of "this deployment needs at least N healthy replicas at all times" — it will happily evict every replica of a deployment if they all happen to be on the node(s) being drained, one drain operation at a time, with no coordination between separate drain operations unless something explicitly enforces it.
The incident: two drains, three minutes apart, zero PDB
- Cluster upgrade automation begins draining nodes one at a time to replace them with a newer node image, a standard rolling node-replacement procedure.
- Node A is drained. It happens to be running 2 of the 3 payments-api replicas. Both are evicted and rescheduled — but only 1 successfully reschedules immediately; the other is briefly pending due to a resource quota edge case. Payments-api is now down to 2 healthy replicas, briefly 1.
- Three minutes later, before the automation's own healthiness check catches up, Node B begins draining — it's running the 3rd replica. It gets evicted too.
- For roughly 45 seconds, payments-api has 0 healthy replicas. Every request to it fails.
The node-drain automation had no way to know that draining Node B would take payments-api to zero replicas, because nothing in the cluster expressed that constraint. It correctly did what a node drain does: evict what's there and move on. The missing piece wasn't better automation — it was a PodDisruptionBudget telling Kubernetes itself "never let this go below 2 healthy replicas, no matter what's asking."
What a PodDisruptionBudget actually enforces
A PDB doesn't prevent evictions in general — it prevents *voluntary* evictions (drains, descheduling, cluster-autoscaler scale-downs) from violating a minimum-availability constraint you define. With a PDB in place, kubectl drain on Node B would have been *blocked* from evicting the last remaining payments-api replica — the drain command would have failed with a clear error instead of silently succeeding and taking the service down.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: payments-api-pdb
namespace: prod
spec:
minAvailable: 2
selector:
matchLabels:
app: payments-apiThis tells Kubernetes: never allow a voluntary eviction (drain, descheduler, autoscaler scale-down) to bring payments-api below 2 healthy replicas. If evicting a pod would violate this, the eviction is rejected — the operation performing the drain gets a clear 429/error instead of silently succeeding.
Why a missing PDB is invisible until exactly this happens
A missing PDB produces zero symptoms during normal operation. There's no warning, no degraded state, nothing in a dashboard that says "this deployment has no disruption protection." It only manifests the one time multiple replicas happen to be co-located on nodes that get drained close together — which might be rare, but cluster upgrades happen regularly, and "rare" events across enough maintenance windows eventually happen.
Relying on every engineer to remember to add a PDB to every new deployment doesn't scale. The better fix — which we adopted afterward — is an admission policy (via OPA Gatekeeper or Kyverno) that rejects any Deployment with more than 1 replica that doesn't have a corresponding PDB. Make the safe thing the default, not an opt-in someone has to remember.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-pdb-for-multi-replica
spec:
validationFailureAction: Enforce
rules:
- name: check-pdb-exists
match:
resources:
kinds: ["Deployment"]
preconditions:
all:
- key: "{{ request.object.spec.replicas }}"
operator: GreaterThan
value: 1
validate:
message: "Deployments with >1 replica must have a matching PodDisruptionBudget."
deny:
conditions:
all:
- key: "{{ length(pdbs) }}"
operator: Equals
value: 0This rejects the Deployment creation itself at admission time if a matching PDB doesn't already exist — turning "remember to add this" into "you cannot forget this," enforced by the cluster.
A 10-minute audit worth running on your own cluster right now
kubectl get deployments -A -o json | jq -r '
.items[] | select(.spec.replicas > 1) | "\(.metadata.namespace)/\(.metadata.name)"
' > all-multi-replica.txt
kubectl get pdb -A -o json | jq -r '
.items[] | .spec.selector.matchLabels | to_entries[] | "\(.key)=\(.value)"
' > all-pdb-selectors.txt
# Manually cross-reference: any deployment in all-multi-replica.txt whose
# labels don't match anything in all-pdb-selectors.txt has no protection.This is a blunt, manual cross-reference — good enough to find the gap quickly. If you find more than a couple of unprotected multi-replica deployments, that's your signal to invest in the admission-policy version instead of fixing them one at a time.