Trace Debugging Workflows

Debugging Workflow

Trace debugging follows a systematic path: identify the slow or failing request, follow the critical path through services, find the bottleneck span, and analyze its attributes and events. This workflow applies to both latency and error investigations.

1. Find the trace from a metric exemplar, log entry, or direct ID.
2. Identify the critical path (longest chain of spans).
3. Look for spans with error status or unusual duration.
4. Check span attributes for error details.
5. Examine downstream dependency responses.
6. Review retry attempts and their outcomes.

Critical Path Analysis

The critical path is the sequence of spans that determines the total request duration. It is the longest chain of dependent operations. Identifying the critical path shows which service or operation is the bottleneck.

# Find traces where the database span is the bottleneck
{resource.service.name="api-gateway"} | select(duration > 500ms)

# Find traces with specific slow operations
{span.db.system="postgresql" && duration > 200ms}

Dependency Fanout

Fanout occurs when a service calls multiple downstream services in parallel. The trace shows a tree of child spans. The total duration is determined by the slowest child. Identify the slowest child to find the bottleneck.

api-gateway (100ms total)
  ├── user-service (20ms)     -- fast
  ├── product-service (80ms)  -- bottleneck!
  │   └── database (70ms)     -- root cause
  └── recommendation-service (30ms) -- fast

Retry Detection

Retries appear as repeated spans for the same operation. They increase total duration and can mask the real error. Look for multiple spans with the same name and attributes but different start times.

payment-service (200ms total)
  ├── process-payment (50ms)  -- failed
  ├── process-payment (50ms)  -- retry 1, failed
  └── process-payment (80ms)  -- retry 2, succeeded

Database Span Analysis

Database spans reveal query performance. Check the db.statement attribute for the actual query, db.rows_affected for result size, and duration for execution time. Slow queries often indicate missing indexes or lock contention.

Span: SELECT * FROM orders WHERE user_id = $1
  db.system: postgresql
  db.name: shopdb
  db.statement: SELECT * FROM orders WHERE user_id = $1
  db.rows_affected: 150
  duration: 450ms  -- suspiciously slow!

Downstream Error Causes

When a trace shows errors, follow the error from the failing span to its root cause. The error may be in the span itself, in a downstream service, or in infrastructure. Check span events for error details and stack traces.

api-gateway (error)
  └── payment-service (error)
       └── stripe-api (error)
            Event: "connection timeout after 5000ms"
            Attribute: http.status_code = 504

# Find all error traces for a service
{resource.service.name="payment-service" && status=error}

# Find traces with specific error type
{status=error} | select(span.error.type="ConnectionTimeout")