Stage 5 · Platform
Observability & Autoscaling
Prometheus Operator
ServiceMonitor, PodMonitor, recording rules, alert rules, and remote write.
Prometheus Operator Overview
Prometheus Operator automates Prometheus deployment and configuration. It introduces CRDs: Prometheus (deployment), ServiceMonitor (scrape config), PrometheusRule (alerts/recording rules), and Alertmanager (alert routing). This replaces manually editing prometheus.yml.
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: main
namespace: monitoring
spec:
replicas: 2
retention: 15d
resources:
requests:
cpu: "1"
memory: "4Gi"
limits:
cpu: "2"
memory: "8Gi"
storage:
volumeClaimTemplate:
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 100Gi
serviceMonitorSelector:
matchLabels:
team: platform
ruleSelector:
matchLabels:
team: platform
additionalScrapeConfigs:
- job_name: 'custom-app'
static_configs:
- targets: ['app:9090']The Prometheus resource creates a StatefulSet with the specified replicas, retention, and storage. serviceMonitorSelector and ruleSelector filter which ServiceMonitors and PrometheusRules to load.
ServiceMonitor
ServiceMonitor defines how to scrape metrics from Services. It specifies the label selector to find Services, the port to scrape, the scrape interval, and the metrics path. This is the standard way to add application metrics to Prometheus.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: api-app
namespace: monitoring
labels:
team: platform
spec:
namespaceSelector:
matchNames:
- production
- staging
selector:
matchLabels:
app: api
endpoints:
- port: metrics
interval: 15s
scrapeTimeout: 10s
path: /metrics
honorLabels: true
relabelings:
- sourceLabels: [__meta_kubernetes_pod_label_version]
targetLabel: version
metricRelabelings:
- sourceLabels: [__name__]
regex: '(http_requests_total|http_request_duration_seconds)'
action: keepnamespaceSelector finds Services in production and staging. selector matches Services with label app=api. endpoints specifies the port named 'metrics' to scrape. relabelings add pod labels to metrics. metricRelabelings filter which metrics to keep.
PodMonitor
PodMonitor is similar to ServiceMonitor but targets pods directly. Use it when pods don't have a Service or when you need to scrape pods that aren't service endpoints (e.g., DaemonSets, StatefulSets without headless services).
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
name: node-exporter
namespace: monitoring
labels:
team: platform
spec:
namespaceSelector:
matchNames:
- monitoring
selector:
matchLabels:
app: node-exporter
podMetricsEndpoints:
- port: metrics
interval: 30s
path: /metricsPodMonitor scrapes pods directly without requiring a Service. This is useful for DaemonSets (node-exporter), sidecars, and pods that expose metrics on non-standard ports.
Recording Rules
Recording rules precompute frequently used or expensive PromQL expressions and save the result as a new time series. This improves dashboard performance and reduces Prometheus load. Use recording rules for SLO calculations and complex aggregations.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: slo-recording-rules
namespace: monitoring
labels:
team: platform
spec:
groups:
- name: slo.rules
interval: 30s
rules:
- record: http_requests:rate5m
expr: sum(rate(http_requests_total[5m])) by (service, code)
- record: http_requests:error_rate5m
expr: |
sum(rate(http_requests_total{code=~"5.."}[5m])) by (service)
/
sum(rate(http_requests_total[5m])) by (service)
- record: http_request_duration:p99
expr: |
histogram_quantile(0.99,
sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service)
)Recording rules compute metrics every 30s and store them. The http_requests:error_rate5m metric calculates the 5xx error rate. histogram_quantile computes the p99 latency. Dashboards query these precomputed metrics instead of computing them on the fly.
Alert Rules
Alert rules define conditions that trigger alerts to Alertmanager. Alertmanager routes alerts to Slack, PagerDuty, email, or other receivers. Alert rules use PromQL expressions with for duration to prevent flapping.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: slo-alerts
namespace: monitoring
spec:
groups:
- name: slo.alerts
rules:
- alert: HighErrorRate
expr: |
http_requests:error_rate5m > 0.05
for: 5m
labels:
severity: critical
annotations:
summary: "High error rate for {{ $labels.service }}"
description: "Error rate is {{ $value | humanizePercentage }} (threshold: 5%)"
- alert: HighLatency
expr: |
http_request_duration:p99 > 0.5
for: 10m
labels:
severity: warning
annotations:
summary: "High p99 latency for {{ $labels.service }}"
description: "p99 latency is {{ $value }}s (threshold: 500ms)"HighErrorRate fires when the error rate exceeds 5% for 5 minutes. HighLatency fires when p99 latency exceeds 500ms for 10 minutes. The for duration prevents alerts from firing on brief spikes.
Remote Write
Remote write sends Prometheus metrics to a long-term storage system (Thanos, Cortex, Mimir, or cloud-managed Prometheus). This provides unlimited retention, global querying across clusters, and better scaling than local Prometheus alone.
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: main
spec:
remoteWrite:
- url: "https://mimir.example.com/api/v1/push"
sendExemplars: true
headers:
X-Scope-OrgID: "production"
writeRelabelConfigs:
- sourceLabels: [__name__]
regex: '(http_.*|go_.*|process_.*)'
action: keep
queueConfig:
capacity: 10000
maxSamplesPerSend: 5000
batchSendDeadline: 30s
maxBackoff: 5sRemote write sends a snapshot of all metrics every scrape interval. writeRelabelConfigs filter which metrics to send. queueConfig manages the send queue — capacity prevents data loss during backlogs.
Use Prometheus Operator's Thanos sidecar for long-term storage instead of remote write. Thanos provides deduplication, downsampling, and global query view. Deploy Thanos Sidecar with the Prometheus resource for a complete observability stack.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.