DevOps

Why Our CI Pipeline Took 40 Minutes and What We Actually Cut to Get to 8

No magic caching trick fixed this. It was four separate, boring bottlenecks — and finding them required actually timing every step instead of guessing.

March 22, 20268 min read

CI/CDGitHub Actionsbuild performancedeveloper experience

40 minutes from push to a merged PR having a green check. That was our pipeline six months ago. Nobody could tell me exactly why — the general answer was always "tests are slow" or "it's just a big monorepo." Both were true and both were almost irrelevant. The real breakdown, once I actually timed every single step instead of accepting the vague consensus, looked nothing like what anyone assumed.

Step one: stop guessing, start timing

Before changing anything, I added timing output to every discrete step in the workflow and pulled 20 recent runs. The actual breakdown surprised almost everyone on the team, including me.

Step	Time (before)	% of total
Checkout + install dependencies	9 min	22%
Docker image build (full, no cache)	14 min	35%
Test suite (sequential, single runner)	11 min	27%
Lint + type check	4 min	10%
Deploy/artifact upload	2 min	5%
Total	40 min	100%

Nobody suspected the Docker build

The team consensus was "tests are slow." Tests were 27% of the time. The Docker image build — rebuilding from scratch on every single run with zero layer caching — was 35%, and nobody had actually measured it because it happened silently in a step nobody watched closely.

Fix #1: Docker layer caching (14 min → 3 min)

We were building the image from a cold cache on every run — no --cache-from, no registry cache, nothing persisted between runs. Every single build reinstalled every OS package and every dependency layer from scratch, every time, even when only application code had changed.

Before: no caching at all

- name: Build image
  run: docker build -t app:latest .

This rebuilds every layer from zero on every single CI run — dependencies, OS packages, everything.

After: registry-backed layer cache via Buildx

- uses: docker/setup-buildx-action@v3

- uses: docker/build-push-action@v6
  with:
    context: .
    push: true
    tags: ghcr.io/org/app:${{ github.sha }}
    cache-from: type=registry,ref=ghcr.io/org/app:buildcache
    cache-to: type=registry,ref=ghcr.io/org/app:buildcache,mode=max

Buildx pulls the previous build's layers from the registry cache first. Only layers that actually changed (usually just the final "copy app code" layer) get rebuilt — everything below it (OS, dependencies) is reused.

This alone cut the build step from 14 minutes to about 3 — the only work left on a typical PR was rebuilding the top layer or two where application code actually changed.

Fix #2: Parallelize the test suite (11 min → 4 min)

We ran the full test suite sequentially on a single runner. GitHub Actions supports matrix jobs natively — we just weren't using them. Splitting by package (this is a Go monorepo with ~14 internal packages) and running 4 shards in parallel cut wall-clock test time to roughly the slowest single shard instead of the sum of all of them.

Sharded test matrix

strategy:
  matrix:
    shard: [1, 2, 3, 4]
steps:
  - run: go test ./... -run . -shard=${{ matrix.shard }}/4

4 shards running concurrently on 4 separate runners instead of 1 runner working through everything sequentially — wall-clock time drops roughly 4x, cost stays close to flat since you're paying for the same total compute either way.

Fix #3: Actually cache dependencies correctly (9 min → 90 sec)

We had a dependency cache configured — but keyed only on the branch name, not on the lockfile hash. Every push invalidated it, so it was effectively caching nothing. actions/cache needs a key that changes exactly when the thing you're caching changes, not more, not less.

Before: cache key too broad, invalidates on every push

- uses: actions/cache@v4
  with:
    path: ~/go/pkg/mod
    key: ${{ runner.os }}-${{ github.ref }}

github.ref changes on every push to the branch — this cache key was almost never a hit.

After: keyed on the lockfile hash

- uses: actions/cache@v4
  with:
    path: ~/go/pkg/mod
    key: ${{ runner.os }}-gomod-${{ hashFiles('go.sum') }}
    restore-keys: |
      ${{ runner.os }}-gomod-

Now the cache only invalidates when go.sum actually changes — which is rare. Most PRs get a full cache hit and skip re-downloading dependencies entirely.

Fix #4: Don't run everything on every PR

A docs-only PR was running the full Docker build, the full test matrix, and a deploy-preview step — none of which touch anything a docs change affects. We added path filters so jobs only run when relevant files change.

Path-based job skipping

on:
  pull_request:
    paths-ignore:
      - 'docs/**'
      - '**.md'

A README-only PR now finishes in under a minute (just lint + a lightweight docs check) instead of running the entire 40-minute pipeline for zero code risk.

The result

Step	Before	After
Checkout + install	9 min	1.5 min
Docker build	14 min	3 min
Tests (parallelized)	11 min	4 min (parallel wall-clock)
Lint + type check	4 min	3 min (parallel with tests)
Total (typical PR)	40 min	~8 min

Architecture: sequential (before) vs. parallel + cached (after)

Rendering diagram…

None of this was clever

Every fix here is a standard, well-documented CI feature: registry-backed Docker layer caching, matrix test sharding, correctly-scoped cache keys, and path filters. The hard part wasn't knowing these existed — it was actually timing each step instead of trusting the team's guess about where the time was going. The guess was wrong on the single biggest bottleneck.

If your pipeline feels slow and nobody can point to exactly why, that's the tell. Add real per-step timing before you touch anything — the fix is usually obvious once you can see where the minutes actually are.