Pushgateway Patterns

When to Use Pushgateway

Pushgateway is for short-lived batch jobs that cannot be scraped. Cron jobs, CI/CD pipelines, and one-off scripts run and exit before Prometheus can pull metrics. Pushgateway acts as an intermediary that receives pushed metrics and holds them until Prometheus scrapes it.

Architecture

Pushgateway runs as a stateless HTTP server. Jobs push metrics via HTTP POST to /metrics/job/<job_name>/<optional labels>. Prometheus scrapes Pushgateway on its normal /metrics endpoint. Pushgateway exposes all pushed metrics in Prometheus format.

scrape_configs:
  - job_name: "pushgateway"
    honor_labels: true
    static_configs:
      - targets: ["pushgateway:9091"]

Pushing Metrics

Push metrics using curl or client libraries. The URL path encodes the job name and instance label. The body contains metrics in Prometheus exposition format.

echo "batch_job_duration_seconds 45.2" | \
  curl --data-binary @- http://pushgateway:9091/metrics/job/batch_etl/instance/worker-01

from prometheus_client import CollectorRegistry, Gauge, push_to_gateway

registry = CollectorRegistry()
g = Gauge("batch_records_processed", "Records processed", registry=registry)
g.set(15000)
push_to_gateway("pushgateway:9091", job="batch_etl", registry=registry)

Avoiding Stale Metrics

Pushgateway does not expire metrics by default. Old metrics from previous job runs remain visible indefinitely. This creates a common trap where stale metrics pollute dashboards and alerts.

• Push a timestamp with every metric so staleness is detectable.
• Use the Pushgateway cleanup API to delete groups after job completion.
• Set pushgateway --persistence.interval to periodically flush old metrics.
• Add metadata labels to distinguish between job runs.

Grouping Keys

Grouping keys in the URL path define the metric group identity. Two pushes with the same grouping keys overwrite the same group. Different grouping keys create separate groups. Choose grouping keys that uniquely identify a job run.

# Single job, single instance
/metrics/job/backup/instance/db-primary

# Multiple grouping labels
/metrics/job/deploy/stage/production/region/us-east-1

# Delete a specific group
curl -X DELETE http://pushgateway:9091/metrics/job/backup/instance/db-primary