Tail-Based Sampling

Keeping rare error traces with policy groups, latency thresholds, and probabilistic fallbacks.

8 min readObservability & Monitoring StackOperate

Tail Sampling Overview

Tail sampling makes the sampling decision after the entire trace is collected. This allows policies based on the full trace: error status, total duration, specific attributes, or span count. Tail sampling keeps interesting traces while discarding routine ones.

Tail sampling flow


Application -> Collector (receive all) -> Buffer -> Policy Evaluation -> Export or Drop

Policy Types

Policy	Keeps Traces When	Example
status_code	Status matches	ERROR traces
latency	Duration exceeds threshold	Requests > 1s
probabilistic	Random percentage	10% of all traces
string_attribute	Attribute matches value	http.target=/api/payments
rate_limiting	Under rate limit	First 100 traces/second

Composing Policies

Multiple policies work together. The tail sampling processor evaluates all policies and decides based on the policy combination. A trace is kept if any policy matches. This creates a layered sampling strategy.

Composed policy example


processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 200000
    policies:
      # Layer 1: Always keep errors
      - name: errors
        type: status_code
        status_code: {status_codes: [ERROR]}

      # Layer 2: Always keep slow requests
      - name: slow
        type: latency
        latency: {threshold_ms: 1000}

      # Layer 3: Always keep payment traces
      - name: payments
        type: string_attribute
        string_attribute:
          key: http.target
          values: ["/api/v1/payments", "/api/v1/refunds"]

      # Layer 4: Sample 5% of everything else
      - name: default
        type: probabilistic
        probabilistic: {sampling_percentage: 5}

Collector Configuration

Tail sampling requires the Collector to buffer entire traces before making a decision. Configure num_traces and decision_wait to control memory usage. The Collector must receive all spans from a trace before evaluating policies.

Full tail sampling pipeline


receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 5s
  tail_sampling:
    decision_wait: 10s
    num_traces: 200000
    expected_new_traces_per_sec: 1000
    policies:
      - name: errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: probabilistic
        type: probabilistic
        probabilistic: {sampling_percentage: 10}

exporters:
  otlp/tempo:
    endpoint: tempo:4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling, batch]
      exporters: [otlp/tempo]

Scaling Considerations

Tail sampling requires buffering, which increases memory usage.
num_traces controls how many traces are buffered. Size it for your traffic.
decision_wait must be long enough to collect the slowest traces.
Multiple Collector instances need consistent sampling decisions.
Use hash-based routing to send the same trace to the same Collector.

Monitoring Sampling

Sampling monitoring queries


# Traces sampled in
rate(otelcol_processor_tail_sampling_count_traces_sampled_total{sampled="true"}[5m])

# Traces sampled out
rate(otelcol_processor_tail_sampling_count_traces_sampled_total{sampled="false"}[5m])

# Sampling ratio
rate(otelcol_processor_tail_sampling_count_traces_sampled_total{sampled="true"}[5m])
  / rate(otelcol_processor_tail_sampling_count_traces_sampled_total[5m])

Monitor your sampling ratio

Track the ratio of sampled-in to sampled-out traces. If too many traces are being dropped, adjust your policies. If too many are being kept, increase the probabilistic threshold or tighten policy conditions.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.

Stage 6 · Operate

Tracing Practice