Stage 6 · Operate
Metrics with Prometheus
Service Discovery
Discovering targets with Kubernetes SD, Consul, EC2, file_sd_config, and relabeling.
Why Service Discovery?
Static target lists break when infrastructure scales dynamically. In Kubernetes, EC2 autoscaling groups, or Consul clusters, targets change constantly. Service discovery automatically finds, labels, and tracks targets without manual config changes.
Kubernetes Service Discovery
Kubernetes service discovery watches the API server and automatically generates targets for pods, services, endpoints, nodes, and ingresses. Each discovered target gets meta labels like __meta_kubernetes_namespace and __meta_kubernetes_pod_name.
scrape_configs:
- job_name: "kubernetes-pods"
kubernetes_sd_configs:
- role: pod
namespaces:
names:
- production
- staging
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: "true"
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
target_label: __address__
regex: (.+)
replacement: "$1"
- source_labels: [__meta_kubernetes_namespace]
target_label: namespace
- source_labels: [__meta_kubernetes_pod_name]
target_label: podAdd prometheus.io/scrape: 'true' and prometheus.io/port: '8080' annotations to your pods. Prometheus uses these annotations to decide whether to scrape a pod and which port to use.
Consul Service Discovery
Consul service discovery queries a Consul catalog for registered services. Each service gets meta labels like __meta_consul_service_name and __meta_consul_tags. This works well for non-Kubernetes environments with service registries.
scrape_configs:
- job_name: "consul-services"
consul_sd_configs:
- server: "consul.service.consul:8500"
tags:
- "prometheus"
relabel_configs:
- source_labels: [__meta_consul_service]
target_label: service
- source_labels: [__meta_consul_tags]
regex: ",(?:[^,]+,)*prod(?:,[^,]*)*"
action: keepEC2 Service Discovery
EC2 service discovery queries the AWS API for instances in your account. Each instance gets meta labels from EC2 tags, instance type, availability zone, and VPC. This enables labeling targets by team, environment, or service from EC2 metadata.
scrape_configs:
- job_name: "ec2-instances"
ec2_sd_configs:
- region: us-east-1
access_key: "envvar://AWS_ACCESS_KEY_ID"
secret_key: "envvar://AWS_SECRET_ACCESS_KEY"
port: 9100
filters:
- name: tag:Environment
values:
- production
relabel_configs:
- source_labels: [__meta_ec2_tag_Name]
target_label: name
- source_labels: [__meta_ec2_availability_zone]
target_label: azFile-Based Service Discovery
File-based service discovery reads targets from JSON or YAML files on disk. An external process writes the file, and Prometheus watches for changes. This is useful for custom discovery mechanisms that do not have native SD support.
[
{
"targets": ["10.0.0.1:9100", "10.0.0.2:9100"],
"labels": {
"job": "node-exporter",
"environment": "production",
"team": "infra"
}
}
]Relabeling Targets
Relabeling is the mechanism that transforms discovered targets into useful labeled series. Source labels feed into regex matches, and the result replaces or adds target labels. This is where you control what gets scraped and how it appears in Prometheus.
The keep action retains only targets matching a regex. The drop action removes matching targets. These are the most powerful relabel actions for filtering what Prometheus scrapes. Always prefer dropping unwanted targets over ignoring them at query time.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.