Tempo and Jaeger Tracing

Tempo Overview

Grafana Tempo is a distributed tracing backend optimized for cost and scale. It stores traces in object storage (S3, GCS, Azure Blob) and does not index trace content. Tempo queries traces by ID, making it extremely cost-efficient.

Jaeger Overview

Jaeger is a CNCF distributed tracing system. It indexes span attributes, enabling powerful search across services, operations, and tags. Jaeger is more feature-rich for trace search but requires more storage and operational overhead.

# Docker compose for Jaeger all-in-one
services:
  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "16686:16686"   # UI
      - "4317:4317"     # OTLP gRPC
      - "4318:4318"     # OTLP HTTP
    environment:
      - COLLECTOR_OTLP_ENABLED=true

OTLP Ingestion

Both Tempo and Jaeger accept traces via OTLP (OpenTelemetry Protocol). OTLP is the standard protocol for sending traces from OpenTelemetry SDKs to backends. It supports both gRPC and HTTP transports.

# Tempo config for OTLP ingestion
server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

Tempo Architecture

Tempo uses a microservices architecture: distributors receive spans, ingesters batch and flush to storage, queriers search traces, and compactors merge storage blocks. For small deployments, the monolithic mode runs all components in a single process.

# Monolithic mode for small deployments
server:
  http_listen_port: 3200

storage:
  trace:
    backend: local
    local:
      path: /var/tempo/traces
    wal:
      path: /var/tempo/wal

metrics_generator:
  registry:
    external_labels:
      source: tempo
      cluster: production
  storage:
    path: /var/tempo/generator/wal
  ring:
    kvstore:
      store: inmemory

Retention Policies

Configure retention to control trace storage costs. Tempo automatically deletes traces older than the retention period. Use tail sampling to reduce ingestion volume while keeping interesting traces.

compactor:
  compaction:
    block_retention: 48h    # 2 days

# For longer retention with object storage
storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-traces
      endpoint: s3.us-east-1.amazonaws.com
    blocklist_poll: 5m

Choosing a Backend