Stage 5 · Platform
Operators, CRDs & Controllers
Controller Observability
Reconcile metrics, structured logs, Kubernetes events, pprof, and stuck finalizer alerts.
Controller Metrics
Controller-runtime exposes metrics: reconcile count, reconcile errors, reconcile duration, and work queue depth. These metrics reveal controller health: error rates, processing latency, and queue backlog.
# Key metrics
controller_runtime_reconcile_total{controller="webapp",result="success"}
controller_runtime_reconcile_total{controller="webapp",result="error"}
controller_runtime_reconcile_time_seconds_bucket{controller="webapp"}
controller_runtime_max_concurrent_reconciles{controller="webapp"}
workqueue_depth{name="webapp"}
workqueue_adds_total{name="webapp"}
workqueue_queue_duration_seconds_bucket{name="webapp"}
---
# Prometheus alert rules
- alert: ControllerHighErrorRate
expr: |
rate(controller_runtime_reconcile_total{result="error"}[5m])
/ rate(controller_runtime_reconcile_total[5m]) > 0.05
for: 5m
labels:
severity: critical
annotations:
summary: "Controller {{ $labels.controller }} has high error rate"
- alert: ControllerReconcileSlow
expr: |
histogram_quantile(0.99,
rate(controller_runtime_reconcile_time_seconds_bucket[5m])
) > 10
for: 5m
labels:
severity: warning
annotations:
summary: "Controller {{ $labels.controller }} reconcile is slow"controller_runtime_reconcile_total counts successes and errors. error rate > 5% indicates a problem. reconcile time p99 > 10s indicates slow reconciliation. workqueue_depth > 0 means the controller is behind.
Structured Logging
Structured logs use JSON format with consistent fields: controller, resource, namespace, name, and result. This enables efficient querying and alerting. Use controller-runtime's logr for structured logging.
Kubernetes Events
Kubernetes Events record significant state changes. Controllers emit events for creation, updates, errors, and conditions. Events are visible in kubectl describe and can be exported for monitoring.
pprof Profiling
pprof provides Go runtime profiling: CPU, memory, goroutines, and blocking. Enable pprof on the metrics endpoint to diagnose performance issues in controllers.
Stuck Finalizer Alerts
Stuck finalizers prevent resources from being deleted. Alert when a resource has a finalizer and DeletionTimestamp is set for too long. This indicates the cleanup logic is failing.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: controller-alerts
spec:
groups:
- name: controller.rules
rules:
- alert: StuckFinalizer
expr: |
kube_customresource_status_condition{
condition="DeletionTimestamp",
status="True"
} > 0
for: 30m
labels:
severity: warning
annotations:
summary: "Resource {{ $labels.name }} stuck with finalizer"
- alert: HighReconcileErrors
expr: |
sum(rate(controller_runtime_reconcile_total{result="error"}[5m])) by (controller)
> 0.1
for: 10m
labels:
severity: critical
annotations:
summary: "Controller {{ $labels.controller }} has sustained errors"StuckFinalizer fires when a resource has been deleting for 30+ minutes with a finalizer. HighReconcileErrors fires when the error rate exceeds 10% for 10 minutes. Both indicate controller issues.
Controller Dashboards
# Reconcile rate
expr: sum(rate(controller_runtime_reconcile_total[5m])) by (controller)
# Error rate
expr: |
sum(rate(controller_runtime_reconcile_total{result="error"}[5m])) by (controller)
/
sum(rate(controller_runtime_reconcile_total[5m])) by (controller)
# Reconcile latency (p99)
expr: |
histogram_quantile(0.99,
sum(rate(controller_runtime_reconcile_time_seconds_bucket[5m])) by (le, controller)
)
# Work queue depth
expr: sum(workqueue_depth) by (name)Dashboard panels: reconcile rate (throughput), error rate (health), p99 latency (performance), and queue depth (backlog). These four panels give a complete view of controller health.
Monitor: reconcile error rate (< 1%), reconcile latency (p99 < 5s), queue depth (0), goroutine count (stable), and memory usage (no leaks). Set alerts for each metric.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.