Collector Operations

Monitoring the Collector

The OTel Collector exposes its own metrics on port 8888. Monitor these metrics to ensure the Collector is healthy. Key metrics include queue size, batch sizes, export failures, and memory usage.

# Collector exposes metrics at :8888/metrics
# Add a scrape config to Prometheus
scrape_configs:
  - job_name: "otel-collector"
    static_configs:
      - targets: ["otel-collector:8888"]

Queue Metrics

The exporter queue holds batches of telemetry waiting to be sent. If the backend cannot keep up, the queue fills up. When the queue is full, new telemetry is dropped. Monitor queue size and drops to detect backend lag.

# Queue size
otelcol_exporter_queue_size

# Queue capacity
otelcol_exporter_queue_capacity

# Items sent per second
rate(otelcol_exporter_sent_spans_total[5m])

# Items dropped (queue full)
rate(otelcol_exporter_send_failed_spans_total[5m])

Memory Limiter

The memory_limiter processor prevents the Collector from exceeding memory limits. When memory usage exceeds the limit, the processor drops incoming telemetry and returns errors to senders. This protects the Collector from OOM kills.

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512         # Hard limit
    spike_limit_mib: 128   # Buffer for spikes
    # Total memory usage triggers dropping
    # when limit_mib is exceeded

# Memory limit
otelcol_processor_memory_limiter_limit_mib

# Current memory usage
otelcol_process_runtime_alloc_bytes

# Dropped spans due to memory pressure
rate(otelcol_processor_memory_limiter_dropped_spans_total[5m])

Batch Processor

The batch processor groups spans into batches before sending. This reduces network overhead and improves throughput. Configure batch size and timeout to balance latency and efficiency.

processors:
  batch:
    timeout: 5s              # Max wait before sending
    send_batch_size: 1000    # Max spans per batch
    send_batch_max_size: 2000  # Absolute max per batch

Retry Settings

Configure retry behavior for failed exports. Retries prevent data loss during temporary backend outages. Exponential backoff prevents overwhelming the backend during recovery.

exporters:
  otlp:
    endpoint: tempo:4317
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s

Handling Exporter Failures

• Monitor otelcol_exporter_send_failed_spans_total for drop rates.
• Alert when drops exceed a threshold over a 5-minute window.
• Check backend health when drops spike.
• Increase queue capacity if drops are due to transient backend slowness.
• Add a fallback exporter to a second backend for critical telemetry.