Step 01 of 06
Understand the idea
Beacon can check other websites, but it does not yet report its own health. This course adds metrics, dashboards, alerts, and incident practices so an operator can understand how Beacon is working.
Key technical ideas
Monitoring starts with questions you already know
Monitoring checks known conditions, such as whether a process is running or errors are increasing. Observability helps you investigate a problem you did not predict, such as slow requests in one region. Services need both: monitoring can alert you, and logs, metrics, and traces can help explain what happened.
Metrics summarize groups of requests
A metric such as beacon_checks_total{result="ok"} stores a running count for that label set. It does not store a separate record for every request. This keeps metrics compact, but they cannot tell you the details of one specific request. Use logs or traces for that. Metrics are best for questions about groups of requests, such as rate, error ratio, and latency.
Metric labels can create too many time series
Every unique combination of metric labels creates a separate time series. If a metric has 5 regions and 4,000 endpoints, it can create 20,000 series. Adding a label with a different value for every user can create far more. This growth is called cardinality. Avoid labels with many possible values, such as user IDs.
Prometheus collects metrics by scraping
Prometheus usually collects metrics by sending an HTTP request to each target; this is called scraping. If a scrape fails, Prometheus records that the target is down. Short-lived jobs may need to push their metrics because they finish before the next scrape. Beacon runs continuously, so scraping is a good fit.
metric: checkout_requests_total
labels: method=2, status=5, route=12
series: 2 * 5 * 12 = 120
add store_id=800
series: 120 * 800 = 96,000
add customer_id=2,000,000
series: 96,000 * 2,000,000 = 192,000,000,000The metric name did not change. The code may have added one innocent label. The storage bill and query cost changed by nine orders of magnitude. That is why SREs ask about cardinality before they ask about dashboard colours.
The three signals do different work
| Signal | Best at | Cost trap |
|---|---|---|
Logs | One event with context | Higlab volume and retention |
Metrics | Trends, ratios, alerts | High-cardinality labels |
Traces | Cross-service latency paths | Sampling and storage cost |
Profiles | CPU and memory hotspots | Usually not always-on |
Technical terms used at work
Time series — One stream of samples for one metric name and one exact set of labels.
Cardinality — The count of distinct label combinations a metric can produce.
Scrape — One Prometheus HTTP collection attempt against a target's metrics endpoint.
Known-unknown — A question you expected to ask, such as whether error rate is above 1%.
