Course outline

01 · Kubernetes Architecture Deep Dive

02 · Workloads & Scheduling

03 · Networking, Storage & Service Mesh

04 · Platform Engineering & Multi-Tenancy

05 · Networking & Service Mesh

06 · Observability & Autoscaling

07 · Configuration & Secrets Management

08 · Stateful Storage Operations

09 · Security, Identity & Admission

10 · Release Packaging with Helm & Kustomize

11 · Operators, CRDs & Controllers

12 · Production Troubleshooting & Operations

Stage 5 · Platform

Production Troubleshooting & Operations

Node NotReady Recovery

kubelet logs, CNI failures, disk pressure, taints, leases, and cordon-drain workflows.

8 min readKubernetes & Cloud NativePlatform

NotReady Causes

Node NotReady means kubelet is not reporting to the API server. Common causes: kubelet crash, network partition, disk pressure, memory pressure, PID pressure, and CNI plugin failure.

Check node conditions


# Check node conditions
kubectl describe node worker-1 | grep -A 20 Conditions

# Key conditions to check:
# Ready: kubelet communicating with API server
# DiskPressure: low disk space
# MemoryPressure: low memory
# PIDPressure: too many processes
# NetworkUnavailable: CNI not ready

Each condition has a status (True/False/Unknown) and a last transition time. True conditions indicate the problem is active. Check the message field for details.

kubelet Logs

kubelet logs reveal why the node is NotReady. Check for: connection errors to API server, CNI plugin failures, certificate issues, and runtime errors.

kubelet log commands


# Check kubelet status
systemctl status kubelet

# View kubelet logs
journalctl -u kubelet --since "1 hour ago" --no-pager | tail -100

# Check kubelet configuration
cat /var/lib/kubelet/config.yaml

# Check kubelet certificates
ls -la /var/lib/kubelet/pki/

# Check kubelet health
curl -k https://localhost:10250/healthz

kubelet logs show: connection refused (kubelet down), x509 certificate errors (certificate expired), CNI plugin errors (network not ready), and OOM kills (memory pressure).

CNI Failures

CNI failures prevent pods from getting network connectivity. Common issues: CNI binary missing, CNI config invalid, CNI plugin crash, and network plugin not ready.

CNI debugging


# Check CNI plugin pods
kubectl get pods -n kube-system -l k8s-app=calico-node
kubectl get pods -n kube-system -l k8s-app=cilium

# Check CNI configuration on node
ls /etc/cni/net.d/
cat /etc/cni/net.d/10-calico.conflist

# Check CNI binary
ls /opt/cni/bin/

# Check node network readiness
kubectl get node worker-1 -o jsonpath='{.status.conditions[?(@.type=="NetworkUnavailable")]'

CNI failures show as NetworkUnavailable: True. Check: CNI plugin pods are running, CNI config files exist and are valid, CNI binaries are installed, and node has network connectivity.

Disk Pressure

Disk pressure occurs when available disk space drops below the threshold. kubelet evicts pods to free space. Check /var/lib/kubelet, /var/log, and emptyDir usage.

Disk pressure debugging


# Check disk usage on node
df -h /var/lib/kubelet
df -h /var/log

# Check which pods use the most disk
kubectl exec my-pod -- du -sh /var/log/* 2>/dev/null | sort -rh | head

# Check emptyDir usage
kubectl get pods -o json | jq '.items[] |
  select(.spec.volumes[]?.emptyDir != null) |
  .metadata.name'

# Clean up disk space
ssh worker-1 "docker system prune -f"
ssh worker-1 "journalctl --vacuum-size=1G"

Disk pressure is common with logging and temporary files. Check: container logs (kubectl logs --previous), emptyDir usage, and image cache. Prune unused images and containers.

Cordon-Drain Workflow

Cordon marks a node as unschedulable. Drain evicts pods and prepares the node for maintenance. Always cordon before draining. Uncordon after maintenance is complete.

Cordon-drain workflow


# Mark node as unschedulable
kubectl cordon worker-1

# Drain node (evict pods, respect PDBs)
kubectl drain worker-1 \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --grace-period=60 \
  --timeout=300s

# Perform maintenance...

# Uncordon node (allow scheduling again)
kubectl uncordon worker-1

# Verify node is Ready
kubectl get node worker-1

--ignore-daemonsets keeps system pods running. --delete-emptydir-data removes pods using emptyDir. --grace-period gives pods time to shut down. --timeout prevents hanging if PDBs prevent eviction.

Node Recovery

Node recovery steps: 1) Cordon the node, 2) Drain pods, 3) Fix the issue, 4) Uncordon, 5) Verify pods reschedule. If the node cannot be recovered, delete it and let the node group replace it.

Node recovery steps


# 1. Cordon
kubectl cordon worker-1

# 2. Drain
kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data

# 3. SSH and fix
ssh worker-1
systemctl restart kubelet
systemctl status kubelet

# 4. Uncordon
kubectl uncordon worker-1

# 5. Verify
kubectl get node worker-1
kubectl get pods -o wide | grep worker-1

# If node cannot be recovered:
kubectl delete node worker-1
# The node group (ASG/MIG) will provision a replacement

Recovery follows: cordon -> drain -> fix -> uncordon -> verify. If the node is unrecoverable, delete it. The cloud provider's node group provisions a replacement. Pods automatically reschedule.

Node health monitoring

Monitor node conditions with kube_node_status_condition. Alert on: Ready=False, DiskPressure=True, MemoryPressure=True, and PIDPressure=True. Use node-exporter for detailed node metrics.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.