Stage 5 · Platform
Production Troubleshooting & Operations
CrashLoopBackOff Analysis
Exit codes, termination messages, probes, OOMKilled events, and restart backoff timing.
CrashLoopBackOff Overview
CrashLoopBackOff means a container is repeatedly crashing and restarting. kubelet applies exponential backoff between restarts. Common causes: application errors, configuration issues, missing dependencies, OOMKilled, and failed probes.
Container starts
-> Application crashes (exit code non-zero)
-> kubelet restarts after 10s
-> Application crashes again
-> kubelet restarts after 20s
-> Application crashes again
-> kubelet restarts after 40s
-> ... (exponential backoff up to 5 minutes)
-> Pod status: CrashLoopBackOffThe backoff sequence is: 10s, 20s, 40s, 80s, 160s, 300s (capped at 5 minutes). After a container runs for 10 minutes without failing, the backoff resets.
Exit Codes
Exit codes indicate why the container crashed. Common codes: 0 (success — container exited normally), 1 (application error), 137 (OOMKilled or SIGKILL), 139 (SIGSEGV — segmentation fault), 143 (SIGTERM — graceful shutdown).
# Get container exit code
kubectl get pod my-pod -o jsonpath='{.status.containerStatuses[0].lastState.terminated.exitCode}'
# Get termination reason
kubectl get pod my-pod -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'
# Get termination message
kubectl get pod my-pod -o jsonpath='{.status.containerStatuses[0].lastState.terminated.message}'
# Check all container statuses
kubectl describe pod my-pod | grep -A 10 "Container Status"exitCode 137 with reason OOMKilled means the container exceeded its memory limit. exitCode 1 with a message means application error. exitCode 139 means segmentation fault — likely a bug.
Termination Messages
Termination messages provide error details when a container crashes. They are written to /dev/termination-log and visible in kubectl describe. Applications can write custom termination messages.
apiVersion: v1
kind: Pod
metadata:
name: app-with-message
spec:
containers:
- name: app
image: my-app:1.0
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: FallbackToLogsOnError
lifecycle:
preStop:
exec:
- /bin/sh
- -c
- "echo 'Graceful shutdown' > /dev/termination-log && sleep 15terminationMessagePolicy: FallbackToLogsOnError uses the termination log first, and falls back to container logs if the log is empty. This provides debugging information when the container crashes.
OOMKilled
OOMKilled means the container exceeded its memory limit and was killed by the kernel OOM killer. Common causes: memory leaks, insufficient limits, and sudden traffic spikes.
# Check if OOMKilled
kubectl get pod my-pod -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'
# Check memory usage vs limits
kubectl top pod my-pod --containers
# Check node memory pressure
kubectl describe node worker-1 | grep -A 5 "Conditions"
# Check resource requests/limits
kubectl get pod my-pod -o jsonpath='{.spec.containers[0].resources}'If OOMKilled: 1) Increase memory limits, 2) Fix memory leaks in the application, 3) Check if the application needs more memory than expected, 4) Use VPA to right-size memory requests.
Probe Failures
Failed liveness probes cause container restarts. If the liveness probe fails, kubelet kills and restarts the container. Common issues: probe path is wrong, application is slow to start, or application is genuinely unhealthy.
# Check probe results
kubectl describe pod my-pod | grep -A 5 "Liveness"
# Check if probe endpoint is accessible
kubectl exec my-pod -- curl -s http://localhost:8080/healthz
# Check application logs around probe failures
kubectl logs my-pod --since=5m | grep -i "health"
# Verify probe configuration
kubectl get pod my-pod -o yaml | grep -A 10 "livenessProbe"If the liveness probe fails, check: 1) Is the probe path correct? 2) Is the application responding? 3) Is the application slow to start (increase initialDelaySeconds)? 4) Is the probe timeout too short?
Restart Backoff
Restart backoff increases with each failure. The sequence: 10s, 20s, 40s, 80s, 160s, 300s. After 10 minutes of successful running, the backoff resets. Understanding this helps predict when a pod will recover.
Restart 1: 10 seconds
Restart 2: 20 seconds
Restart 3: 40 seconds
Restart 4: 80 seconds
Restart 5: 160 seconds
Restart 6+: 300 seconds (5 minutes, capped)
After 10 minutes of successful run:
Backoff resets to 10 secondsThe backoff is per-container, not per-pod. If a container runs for 10 minutes without failing, the backoff resets. This prevents permanent backoff after transient issues.
1) kubectl describe pod (check events and exit codes), 2) kubectl logs --previous (check crash logs), 3) Check resource limits (OOMKilled?), 4) Check probes (liveness failures?), 5) Check configuration (missing ConfigMap/Secret?).
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.