Observability (8%)

The Three Pillars of Observability

Observability is the ability to understand the internal state of a system by examining its external outputs. In cloud native systems, this means three complementary signal types: metrics, logs, and traces.

Metrics with Prometheus

Prometheus is the CNCF-standard metrics system for Kubernetes. It scrapes metrics endpoints, stores time-series data, and evaluates alerting rules. The PromQL query language is used for analysis and dashboarding.

# CPU usage per pod
rate(container_cpu_usage_seconds_total{namespace="default"}[5m])

# Memory usage percentage
container_memory_working_set_bytes / container_spec_memory_limit_bytes

# HTTP request rate
rate(http_requests_total[5m])

# 99th percentile latency
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))

Logging

In Kubernetes, container logs are written to stdout and stderr. The kubelet captures these and stores them on the node. Aggregation tools like Fluentd, Fluent Bit, or Vector collect logs from nodes and ship them to centralized backends like Elasticsearch or Loki.

• kubectl logs — Direct access to container logs.
• Fluentd — CNCF-standard log collector and forwarder.
• Fluent Bit — Lightweight alternative to Fluentd.
• Loki — Log aggregation by Grafana Labs, Prometheus-like queries.

Distributed Tracing

Tracing follows a request as it flows through multiple services. Each service creates a span, and spans are linked into a trace. This reveals latency bottlenecks and failure points in distributed systems.

• OpenTelemetry — Vendor-neutral instrumentation standard.
• Jaeger — CNCF-distributed tracing system.
• Tempo — Grafana-based tracing backend.
• Zipkin — Twitter-originated tracing system.

SLOs, SLIs, and SLAs

Service Level Indicators (SLIs) are the metrics you measure. Service Level Objectives (SLOs) are the targets you set. Service Level Agreements (SLAs) are the contractual commitments with consequences for failure.

Cost Management

Cloud native costs can spiral without governance. Kubernetes cost management involves right-sizing resources, using spot instances, implementing resource quotas, and leveraging tools like Kubecost or OpenCost.

kubectl top pods                  # Current resource usage
kubectl top nodes                 # Node resource usage
kubectl describe resourcequota    # Namespace quota status
kubectl get nodes -o custom-columns=NODE:.metadata.name,CPU:.status.capacity.cpu,MEM:.status.capacity.memory

Monitoring Stack Comparison