Stage 3 · Build

Performance, Memory & Debugging

Production Benchmarking

benchstat, realistic fixtures, shadow traffic, and safe profiling in live services.

7 min readGo for SREBuild

benchstat

benchstat compares benchmark results and reports statistical significance. It tells you whether a change is a real improvement or just noise.

benchstat workflow


# Install benchstat
go install golang.org/x/perf/cmd/benchstat@latest

# Save baseline
go test -bench=. -benchmem -count=10 > old.txt

# Make changes, save new results
go test -bench=. -benchmem -count=10 > new.txt

# Compare
benchstat old.txt new.txt

# Output:
# name     old time/op    new time/op    delta
# Parse-8    1.23µs ± 2%    0.98µs ± 1%  -20.33%  (p=0.000 n=10+10)
# Marshal-8  2.45µs ± 3%    2.41µs ± 2%     ~     (p=0.142 n=10+10)

# name     old alloc/op   new alloc/op   delta
# Parse-8      256B ± 0%      128B ± 0%  -50.00%  (p=0.000 n=10+10)
# Marshal-8    512B ± 0%      512B ± 0%     ~     (p=1.000 n=10+10)

# Filter by name
benchstat -filter 'Parse' old.txt new.txt

# HTML output
benchstat -html old.txt new.txt > report.html

benchstat requires multiple runs (count=10) for statistical analysis. The p-value indicates whether the change is significant. p < 0.05 means statistically significant. delta shows the percentage change.

Realistic Fixtures

Benchmarks with synthetic data do not reflect real performance. Use realistic fixtures — data that matches production characteristics in size, complexity, and distribution.

Gorealistic-benchmark-fixtures.go

45 linesLn 1, Col 1Go

Realistic fixtures capture production behavior. Test with different data sizes to understand scaling characteristics. Sub-benchmarks with varying inputs show how performance changes with load. Store fixtures in testdata/ directories.

Benchmark Metrics

Gocustom-benchmark-metrics.go

32 linesLn 1, Col 1Go

b.ReportMetric adds custom metrics to benchmark output. b.ReportAllocs reports allocation counts and bytes. b.Elapsed() returns the benchmark duration. Use these to capture performance characteristics beyond time.

Shadow Traffic

Shadow traffic replays real production requests against a new version. It measures real-world performance without affecting users.

Goshadow-traffic-setup.go

32 linesLn 1, Col 1Go

Shadow traffic copies requests to a new version. The primary response is served normally. The shadow response is logged but not returned to the user. Compare latency, error rates, and resource usage between primary and shadow.

Safe Profiling in Production

Goproduction-safe-profiling.go

37 linesLn 1, Col 1Go

Production profiling must be safe. Use short durations (5-10 seconds). Rate-limit profiling requests. Use localhost-only pprof endpoints. Profile during low-traffic periods. The overhead is small but real — profile sparingly.

Regression Detection

CI benchmark comparison


# GitHub Actions benchmark comparison
name: Benchmark
on:
  pull_request:
    paths:
      - '**/*.go'

jobs:
  benchmark:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-go@v5
        with:
          go-version: "1.22"

      - name: Save baseline
        run: |
          git checkout main
          go test -bench=. -benchmem -count=5 > old.txt

      - name: Save PR results
        run: |
          git checkout -
          go test -bench=. -benchmem -count=5 > new.txt

      - name: Compare
        run: |
          go install golang.org/x/perf/cmd/benchstat@latest
          benchstat old.txt new.txt

      - name: Fail on regression
        run: |
          benchstat -delta 1% old.txt new.txt | grep -q "regression" && exit 1 || true

CI benchmark comparison catches performance regressions before they reach production. Save baseline from main, compare with PR results, and fail on significant regressions. The delta threshold controls sensitivity.

Benchmark regularly

Run benchmarks on every PR. Compare with the baseline. Fail on regressions. This creates a culture of performance awareness. Small regressions accumulate — catch them early when they are easy to fix.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.

Stage 3 · Build

Performance, Memory & Debugging

Production Benchmarking

benchstat, realistic fixtures, shadow traffic, and safe profiling in live services.

7 min readGo for SREBuild

benchstat

benchstat compares benchmark results and reports statistical significance. It tells you whether a change is a real improvement or just noise.

benchstat workflow


# Install benchstat
go install golang.org/x/perf/cmd/benchstat@latest

# Save baseline
go test -bench=. -benchmem -count=10 > old.txt

# Make changes, save new results
go test -bench=. -benchmem -count=10 > new.txt

# Compare
benchstat old.txt new.txt

# Output:
# name     old time/op    new time/op    delta
# Parse-8    1.23µs ± 2%    0.98µs ± 1%  -20.33%  (p=0.000 n=10+10)
# Marshal-8  2.45µs ± 3%    2.41µs ± 2%     ~     (p=0.142 n=10+10)

# name     old alloc/op   new alloc/op   delta
# Parse-8      256B ± 0%      128B ± 0%  -50.00%  (p=0.000 n=10+10)
# Marshal-8    512B ± 0%      512B ± 0%     ~     (p=1.000 n=10+10)

# Filter by name
benchstat -filter 'Parse' old.txt new.txt

# HTML output
benchstat -html old.txt new.txt > report.html

Realistic Fixtures

Benchmarks with synthetic data do not reflect real performance. Use realistic fixtures — data that matches production characteristics in size, complexity, and distribution.

Gorealistic-benchmark-fixtures.go

45 linesLn 1, Col 1Go

Benchmark Metrics

Gocustom-benchmark-metrics.go

32 linesLn 1, Col 1Go

Shadow Traffic

Shadow traffic replays real production requests against a new version. It measures real-world performance without affecting users.

Goshadow-traffic-setup.go

32 linesLn 1, Col 1Go

Safe Profiling in Production

Goproduction-safe-profiling.go

37 linesLn 1, Col 1Go

Regression Detection

CI benchmark comparison


# GitHub Actions benchmark comparison
name: Benchmark
on:
  pull_request:
    paths:
      - '**/*.go'

jobs:
  benchmark:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-go@v5
        with:
          go-version: "1.22"

      - name: Save baseline
        run: |
          git checkout main
          go test -bench=. -benchmem -count=5 > old.txt

      - name: Save PR results
        run: |
          git checkout -
          go test -bench=. -benchmem -count=5 > new.txt

      - name: Compare
        run: |
          go install golang.org/x/perf/cmd/benchstat@latest
          benchstat old.txt new.txt

      - name: Fail on regression
        run: |
          benchstat -delta 1% old.txt new.txt | grep -q "regression" && exit 1 || true

Benchmark regularly

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.