Logs, Metrics, Traces

What Is Observability?

Observability is the ability to understand the internal state of a system from its external outputs. A system is observable if you can diagnose problems without deploying new code. The three pillars — logs, metrics, and traces — provide different views of the same system.

Logs

Logs are discrete events with timestamps. They capture what happened at a specific moment. Logs are the most granular form of observability — they tell you exactly what a system did. Unstructured logs are hard to search; structured logs are machine-readable.

{
  "timestamp": "2025-01-15T10:30:45.123Z",
  "level": "error",
  "service": "payment-api",
  "trace_id": "abc123def456",
  "method": "POST",
  "path": "/api/v1/charge",
  "status": 500,
  "duration_ms": 2340,
  "error": "connection refused to payment-gateway:5432",
  "user_id": "u_98765"
}

Metrics

Metrics are numerical measurements over time. They answer questions about trends: Is latency increasing? Is error rate rising? Is the system saturated? Metrics are compact, efficient, and ideal for dashboards and alerting.

• Counters — Monotonically increasing (request count, error count).
• Gauges — Can go up or down (CPU usage, queue depth, connections).
• Histograms — Distribution of values (request latency, payload size).
• Summaries — Precomputed percentiles (p50, p99 latency).

Traces

Traces follow a single request as it flows through multiple services. Each service creates a span, and spans are linked into a trace. Traces answer: Which service is slow? Where is the bottleneck? What path did the request take?

Structured Logging

Structured logging is the most impactful observability investment. Convert all logs to JSON with consistent fields: timestamp, level, service, trace_id, and domain-specific context. This enables correlation across services and efficient querying.

When to Use What