Distributed Tracing

Why Distributed Tracing?

In a microservices architecture, a single user request may traverse 10-30 services. When that request is slow or fails, you need to know which service caused the problem. Distributed tracing follows the request through every service and shows exactly where time was spent.

OpenTelemetry

OpenTelemetry (OTel) is the CNCF standard for telemetry collection. It provides SDKs for traces, metrics, and logs in every major language. OTel auto-instrumentation libraries capture traces without code changes for popular frameworks and libraries.

# Install: pip install opentelemetry-distro opentelemetry-exporter-otlp
# Run: opentelemetry-instrument python app.py

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider()
processor = BatchSpanProcessor(OTLPSpanExporter())
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

tracer = trace.get_tracer("payment-service")

with tracer.start_as_current_span("process_payment") as span:
    span.set_attribute("payment.amount", 99.99)
    span.set_attribute("payment.currency", "USD")
    # Child spans are automatically linked

Context Propagation

Context propagation is how trace context passes from one service to the next. The trace ID and span ID are injected into HTTP headers (W3C Trace Context: traceparent). The next service extracts the context and creates child spans.

Service A                    Service B                    Service C
┌──────────┐               ┌──────────┐               ┌──────────┐
│ Span: A  │──traceparent──►│ Span: B  │──traceparent──►│ Span: C  │
│ trace: T │  header        │ trace: T │  header        │ trace: T │
│ parent: -│               │ parent: A │               │ parent: B │
└──────────┘               └──────────┘               └──────────┘

All three spans share trace ID T, forming a trace tree:
T
├── A (root)
│   └── B (child of A)
│       └── C (child of B)

Spans and Span Attributes

A span represents a single operation within a trace. Spans have a name, start time, end time, status, and attributes. Attributes are key-value pairs that provide context — HTTP method, database query, user ID, error message.

Sampling Strategies

Tracing generates enormous volumes of data. Sampling controls how many traces are collected. Always-on sampling is too expensive. Always-off loses data. Head-based and tail-based sampling provide balanced approaches.

Tracing Backends