Trace Backends

Backend Comparison

Trace backends differ in storage architecture, query capabilities, and operational complexity. The choice depends on your scale, search requirements, and existing observability stack.

Jaeger Deep Dive

Jaeger indexes span attributes, enabling powerful search across services, operations, tags, and durations. It supports head and tail sampling, dependency graphs, and comparison of traces. Jaeger is more feature-rich for trace investigation.

# Jaeger with Elasticsearch backend
collector:
  image: jaegertracing/jaeger-collector:latest
  environment:
    - SPAN_STORAGE_TYPE=elasticsearch
    - ES_SERVER_URLS=http://elasticsearch:9200
    - ES_INDEX_SHARDS=3
    - ES_INDEX_REPLICAS=1

query:
  image: jaegertracing/jaeger-query:latest
  environment:
    - SPAN_STORAGE_TYPE=elasticsearch
    - ES_SERVER_URLS=http://elasticsearch:9200

Tempo Deep Dive

Tempo stores traces in object storage with minimal indexing. It can only query by trace ID unless you use TraceQL. This makes Tempo extremely cost-efficient but requires correlation with metrics or logs to find traces.

# Find traces with errors
{status=error}

# Find slow traces
{duration > 1s}

# Find traces hitting specific service
{resource.service.name="payment-service"}

# Combine conditions
{resource.service.name="payment-service" && status=error && duration > 500ms}

Zipkin

Zipkin is one of the original distributed tracing systems. It is simpler than Jaeger and Tempo, making it suitable for smaller deployments. Zipkin supports dependency graphs and trace comparison but has fewer query options.

# Zipkin all-in-one
services:
  zipkin:
    image: openzipkin/zipkin:latest
    ports:
      - "9411:9411"
    environment:
      - STORAGE_TYPE=elasticsearch
      - ES_HOSTS=http://elasticsearch:9200

Storage and Indexes

The storage backend determines query capabilities and cost. Elasticsearch indexes all span attributes for full-text search. Object storage (Tempo) indexes only trace ID for minimal cost. Cassandra provides write-optimized storage for high volume.

Query Workflows

• Start from a metric spike and follow an exemplar to a trace.
• Use Grafana Tempo to search by trace ID from logs.
• Use Jaeger to search by service, operation, and tags.
• Use TraceQL to find traces matching specific conditions.
• Compare two traces to identify performance differences.
• View the dependency graph to understand service relationships.