Stage 6 · Operate
Metrics with Prometheus
Prometheus Data Model
Metric names, labels, samples, time series, staleness markers, and scrape timestamps.
Metric Types
Prometheus supports four core metric types, each designed for a different measurement pattern. Choosing the right type is critical for accurate queries and efficient storage.
| Type | Use Case | Example |
|---|---|---|
| Counter | Monotonically increasing value | http_requests_total |
| Gauge | Value that can go up or down | cpu_temperature_celsius |
| Histogram | Sample distribution with buckets | http_request_duration_seconds |
| Summary | Precomputed quantiles on client side | rpc_duration_seconds |
Prefer histograms over summaries in most cases. Histograms allow server-side aggregation across instances, while summaries compute quantiles on the client and cannot be combined meaningfully.
Labels and Time Series
A time series in Prometheus is uniquely identified by a metric name and a set of label name-value pairs. Each unique combination of name and labels defines a distinct time series with its own stream of samples.
http_requests_total{method="GET", endpoint="/api/v1/users", status="200"}
http_requests_total{method="POST", endpoint="/api/v1/users", status="201"}
http_requests_total{method="GET", endpoint="/api/v1/users", status="500"}Each line above is a separate time series. The labels method, endpoint, and status create distinct dimensions. Prometheus stores each series independently, so label cardinality directly affects memory and storage.
Naming Conventions
Prometheus metric names follow strict conventions. The name must include a unit suffix and represent a single measurement type. Following these conventions ensures consistency across teams.
# Counter — total suffix
http_requests_total
disk_reads_total
# Gauge — no suffix or descriptive
heap_size_bytes
cpu_usage_ratio
# Histogram — _seconds, _bytes, etc.
http_request_duration_seconds_bucket
http_request_duration_seconds_sum
http_request_duration_seconds_countLabels like user_id, request_id, or trace_id can create millions of series. Each unique label combination is a separate time series. High cardinality causes memory pressure, slow queries, and increased storage costs.
Scrape Timestamps
When Prometheus scrapes a target, it records the timestamp of the scrape. By default, Prometheus uses the scrape time as the sample timestamp. You can also enable honor_timestamps to use the target's own timestamps.
scrape_configs:
- job_name: "my-app"
honor_timestamps: true
scrape_interval: 15s
static_configs:
- targets: ["localhost:9090"]When honor_timestamps is false (the default), Prometheus replaces the target's timestamp with the time it performed the scrape. This is safer when targets produce stale or incorrect timestamps.
Staleness Markers
Prometheus inserts staleness markers when a previously available series disappears from a scrape. This prevents graphs from showing old data as if it were current. Staleness markers are special NaN values appended after the last sample.
A series becomes stale when its target goes down or stops reporting. The staleness marker is inserted one scrape interval after the last successful scrape. This allows PromQL to correctly handle disappearing series in range queries.
The default staleness window is 5 minutes. For faster detection of disappeared targets, reduce scrape_interval or use the --storage.tsdb.min-block-duration flag. Be aware that very short intervals increase overhead.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.