Platform Observability Stack

Grafana, Tempo, Loki, Mimir/Pyroscope. Distributed traces for platform API. Correlate scaffold → deploy → incident.

8 min readPlatform Engineering & Developer ExperienceMaster

Stack Overview

Platform observability is distinct from application observability. It observes the platform itself: API latency, scaffold success rate, CI queue time, ArgoCD sync status, Crossplane claim status, secret sync status. Uses same tools (Grafana stack) but different dashboards, alerts, and SLIs.

Rendering diagram…

Signal	Tool	Platform Use Case	Retention
Traces	Tempo	Platform API requests, scaffold flow, controller reconcile	7d hot, 90d cold
Logs	Loki	Controller logs, ArgoCD sync logs, scaffold step logs	30d hot, 1y cold
Metrics	Mimir	SLIs (latency, error rate, queue time), controller metrics, queue depths	13mo (for year-over-year)
Profiles	Pyroscope	Platform API CPU/memory, controller hot paths	7d

Platform API Distributed Tracing

Instrument Platform API with OpenTelemetry. Every scaffold request gets a trace: API gateway → auth → validation → scaffolder → GitHub → catalog → ArgoCD → Crossplane. Trace context propagates to child operations.

Goplatform-api-opentelemetry-instrumentation.go

108 linesLn 1, Col 1Go

Platform API handler with OpenTelemetry spans for each major step. Attributes capture business context for querying.

Scaffold → Deploy → Incident Correlation

Correlation IDs flow from scaffold → GitHub → ArgoCD → Kubernetes → incident. Enables: 'Show me all incidents for services scaffolded from template v2.3' or 'What was the scaffold-to-deploy time for this service?'

Correlation ID Propagation


# 1. Scaffold creates operation ID
operationID: "op-abc123"
correlationID: "corr-xyz789"

# 2. Passed to GitHub Actions via environment
env:
  PLATFORM_OPERATION_ID: ${{ steps.scaffold.outputs.operation_id }}
  PLATFORM_CORRELATION_ID: ${{ steps.scaffold.outputs.correlation_id }}

# 3. GitHub Actions passes to ArgoCD via Application annotation
metadata:
  annotations:
    platform.example.com/operation-id: "op-abc123"
    platform.example.com/correlation-id: "corr-xyz789"

# 4. ArgoCD sync creates resources with labels
metadata:
  labels:
    platform.example.com/operation-id: "op-abc123"
    platform.example.com/correlation-id: "corr-xyz789"

# 5. Crossplane claims inherit correlation
metadata:
  labels:
    platform.example.com/correlation-id: "corr-xyz789"

# 5. Incident runbook includes correlation ID
# Runbook: "Check correlation ID in Grafana: tempo query {correlation_id="corr-xyz789"}"

Correlation ID flows through entire pipeline. Enables end-to-end tracing in Tempo/Grafana.

Platform Dashboards

Platform SLI Dashboard: API latency (p50/p95/p99), error rate, scaffold success rate, CI queue time, env provision time
Scaffold Funnel: Requested → Validated → Scaffolded → Provisioned → Registered → Deployed (with drop-off rates)
Controller Health: Reconcile rate, duration, error rate, queue depth per controller
ArgoCD Sync: Sync status, sync duration, drift count, auto-heal count
Crossplane Claims: Claim status, provision time, error rate by type (DB, Redis, Kafka)
Secret Sync: ESO sync rate, secret age, rotation compliance
Cost Attribution: Per-team, per-service, per-environment cost from Kubecost/OpenCost

Platform Alerting

Alert	Condition	Severity	Runbook
Platform API High Latency	p99 > 1s for 5m	Critical	Check API pods, DB, downstream deps
Platform API High Error Rate	5xx > 1% for 5m	Critical	Check logs, recent deploy, dependencies
Scaffold Failure Rate	Failure > 5% for 10m	Critical	Check scaffolder logs, GitHub API, template
CI Queue Time	p95 > 10m for 15m	Warning	Check runner capacity, queue depth
Env Provision Time	p95 > 10m for 15m	Warning	Check Crossplane, cloud provider quotas
ArgoCD Sync Failed	Any app OutOfSync > 10m	Critical	Check ArgoCD app, resource diff
Crossplane Claim Stuck	Claim not Ready > 30m	Warning	Check Crossplane logs, cloud provider
Secret Rotation Overdue	Secret age > TTL + 24h	Warning	Check ESO, Vault, target secret
Controller Reconcile Slow	Reconcile duration p99 > 30s	Warning	Check controller logs, API server
Platform API Down	Health endpoint failing	Critical	Check deployment, ingress, dependencies

Observability Cost Management

Sampling: 10% trace sampling for platform API, 100% for errors
Metrics: Reduce cardinality — no high-cardinality labels (request_id, user_id) in metrics
Logs: Structured JSON, drop debug logs in production, use Loki index labels wisely
Retention: 7d hot (SSD), 90d cold (S3) for traces/logs; 13mo for metrics
Compression: Tempo/Loki/Mimir compression enabled
Cardinality limits: Mimir max_global_series_per_user, Loki max_streams_per_user
Cost attribution: Label all telemetry with team, service, environment for showback

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.

Stage 7 · Master

Observability & Developer Experience Metrics

Platform Observability Stack

Grafana, Tempo, Loki, Mimir/Pyroscope. Distributed traces for platform API. Correlate scaffold → deploy → incident.

8 min readPlatform Engineering & Developer ExperienceMaster

Stack Overview

Platform Observability Stack

Rendering diagram…

Signal	Tool	Platform Use Case	Retention
Traces	Tempo	Platform API requests, scaffold flow, controller reconcile	7d hot, 90d cold
Logs	Loki	Controller logs, ArgoCD sync logs, scaffold step logs	30d hot, 1y cold
Metrics	Mimir	SLIs (latency, error rate, queue time), controller metrics, queue depths	13mo (for year-over-year)
Profiles	Pyroscope	Platform API CPU/memory, controller hot paths	7d

Platform API Distributed Tracing

Goplatform-api-opentelemetry-instrumentation.go

108 linesLn 1, Col 1Go

Platform API handler with OpenTelemetry spans for each major step. Attributes capture business context for querying.

Scaffold → Deploy → Incident Correlation

Correlation ID Propagation


# 1. Scaffold creates operation ID
operationID: "op-abc123"
correlationID: "corr-xyz789"

# 2. Passed to GitHub Actions via environment
env:
  PLATFORM_OPERATION_ID: ${{ steps.scaffold.outputs.operation_id }}
  PLATFORM_CORRELATION_ID: ${{ steps.scaffold.outputs.correlation_id }}

# 3. GitHub Actions passes to ArgoCD via Application annotation
metadata:
  annotations:
    platform.example.com/operation-id: "op-abc123"
    platform.example.com/correlation-id: "corr-xyz789"

# 4. ArgoCD sync creates resources with labels
metadata:
  labels:
    platform.example.com/operation-id: "op-abc123"
    platform.example.com/correlation-id: "corr-xyz789"

# 5. Crossplane claims inherit correlation
metadata:
  labels:
    platform.example.com/correlation-id: "corr-xyz789"

# 5. Incident runbook includes correlation ID
# Runbook: "Check correlation ID in Grafana: tempo query {correlation_id="corr-xyz789"}"

Correlation ID flows through entire pipeline. Enables end-to-end tracing in Tempo/Grafana.

Platform Dashboards

Platform SLI Dashboard: API latency (p50/p95/p99), error rate, scaffold success rate, CI queue time, env provision time
Scaffold Funnel: Requested → Validated → Scaffolded → Provisioned → Registered → Deployed (with drop-off rates)
Controller Health: Reconcile rate, duration, error rate, queue depth per controller
ArgoCD Sync: Sync status, sync duration, drift count, auto-heal count
Crossplane Claims: Claim status, provision time, error rate by type (DB, Redis, Kafka)
Secret Sync: ESO sync rate, secret age, rotation compliance
Cost Attribution: Per-team, per-service, per-environment cost from Kubecost/OpenCost

Platform Alerting

Alert	Condition	Severity	Runbook
Platform API High Latency	p99 > 1s for 5m	Critical	Check API pods, DB, downstream deps
Platform API High Error Rate	5xx > 1% for 5m	Critical	Check logs, recent deploy, dependencies
Scaffold Failure Rate	Failure > 5% for 10m	Critical	Check scaffolder logs, GitHub API, template
CI Queue Time	p95 > 10m for 15m	Warning	Check runner capacity, queue depth
Env Provision Time	p95 > 10m for 15m	Warning	Check Crossplane, cloud provider quotas
ArgoCD Sync Failed	Any app OutOfSync > 10m	Critical	Check ArgoCD app, resource diff
Crossplane Claim Stuck	Claim not Ready > 30m	Warning	Check Crossplane logs, cloud provider
Secret Rotation Overdue	Secret age > TTL + 24h	Warning	Check ESO, Vault, target secret
Controller Reconcile Slow	Reconcile duration p99 > 30s	Warning	Check controller logs, API server
Platform API Down	Health endpoint failing	Critical	Check deployment, ingress, dependencies

Observability Cost Management

Sampling: 10% trace sampling for platform API, 100% for errors
Metrics: Reduce cardinality — no high-cardinality labels (request_id, user_id) in metrics
Logs: Structured JSON, drop debug logs in production, use Loki index labels wisely
Retention: 7d hot (SSD), 90d cold (S3) for traces/logs; 13mo for metrics
Compression: Tempo/Loki/Mimir compression enabled
Cardinality limits: Mimir max_global_series_per_user, Loki max_streams_per_user
Cost attribution: Label all telemetry with team, service, environment for showback

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.