Sampling Strategies

Why Sample?

Tracing every request generates enormous volumes of spans. A service handling 10,000 requests per second produces 10,000+ spans per second per hop. Sampling reduces volume while preserving visibility into interesting requests.

Head Sampling

Head sampling decides at the start of a trace whether to sample it. The decision propagates through context propagation. All services in the trace follow the same decision. This is the simplest approach but cannot make informed decisions about the full trace.

Tail Sampling

Tail sampling collects all spans initially and decides after the trace completes whether to keep it. The OTel Collector's tail_sampling processor evaluates the full trace against policies. This keeps error traces and slow traces while sampling normal ones.

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    expected_new_traces_per_sec: 1000
    policies:
      - name: errors
        type: status_code
        status_code: {status_codes: [ERROR]}

      - name: slow-traces
        type: latency
        latency: {threshold_ms: 1000}

      - name: probabilistic
        type: probabilistic
        probabilistic: {sampling_percentage: 10}

Probabilistic Sampling

Probabilistic sampling keeps a fixed percentage of traces. It is predictable and easy to reason about. The tradeoff is that it may miss important traces (errors, slow requests) in the unsampled portion.

processors:
  probabilistic_sampler:
    sampling_percentage: 10
    sampling_source_key: "http.target"
    decision_server: false

Attribute-Based Sampling

Attribute-based sampling makes decisions based on span attributes. You can always sample traces with errors, traces exceeding a latency threshold, or traces from specific endpoints. This ensures important traces are never dropped.

processors:
  tail_sampling:
    policies:
      # Always keep error traces
      - name: errors
        type: status_code
        status_code: {status_codes: [ERROR]}

      # Always keep traces hitting payment endpoint
      - name: payment
        type: string_attribute
        string_attribute: {key: "http.target", values: ["/api/v1/payments"]}

      # Sample 5% of everything else
      - name: default
        type: probabilistic
        probabilistic: {sampling_percentage: 5}