Stage 5 · Platform
CI/CD Foundations
Pipeline Failure Modes
Diagnosing broken runners, expired credentials, cache poisoning, missing artifacts, and nondeterministic builds.
Runner Failures
Runners are the foundation of your pipeline. When runners fail, everything stops. Common runner failures include disk full, out of memory, network timeouts, and incompatible tool versions. Hosted runners rarely fail, but self-hosted runners require regular maintenance.
jobs:
build:
runs-on: self-hosted
steps:
- name: Check disk space
run: |
DISK_FREE=$(df -BG / | awk 'NR==2 {print $4}' | tr -d 'G')
if [ "$DISK_FREE" -lt 5 ]; then
echo "::error::Insufficient disk space: ${DISK_FREE}GB free"
exit 1
fi
- name: Check memory
run: |
FREE_MEM=$(free -m | awk '/^Mem:/ {print $7}')
if [ "$FREE_MEM" -lt 1024 ]; then
echo "::error::Insufficient memory: ${FREE_MEM}MB free"
exit 1
fi
- uses: actions/checkout@v4
- run: npm ci && npm testPre-flight checks catch runner issues before they waste time on a long build. These checks run in seconds and provide clear error messages when the runner environment is unhealthy.
Long-lived self-hosted runners accumulate state — old processes, full disks, stale caches. Use ephemeral runners that are destroyed after each job. Kubernetes executors and auto-scaling groups make this easy.
Expired Credentials
Expired tokens, rotated secrets, and revoked API keys are among the most common pipeline failures. A pipeline that worked yesterday fails today because a credential expired at midnight. This is especially common with long-lived tokens.
jobs:
deploy:
runs-on: ubuntu-latest
permissions:
id-token: write
contents: read
steps:
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::123456789012:role/deploy
aws-region: us-east-1
- run: aws s3 sync dist/ s3://my-bucket/OIDC tokens are short-lived (minutes) and automatically rotated. The GitHub OIDC provider federates with AWS IAM, eliminating the need for stored AWS access keys. The token exists only during the job and is revoked when the job completes.
Cache Poisoning
Caches speed up pipelines but can introduce subtle bugs. A corrupted cache can cause mysterious failures that are hard to reproduce locally. Cache poisoning — where malicious code modifies cached artifacts — is a supply chain security concern.
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Cache node modules
uses: actions/cache@v4
with:
path: node_modules
key: node-modules-${{ runner.os }}-${{ hashFiles('package-lock.json') }}
restore-keys: |
node-modules-${{ runner.os }}-
- run: npm ci
- run: npm testThe cache key includes a hash of package-lock.json. When dependencies change, a new cache is created with a different key. This prevents stale or corrupted caches from affecting builds. The restore-keys fallback allows partial cache hits.
Nondeterministic Builds
A nondeterministic build produces different outputs from the same input. This can be caused by floating dependencies, timestamps in build output, random test data, or race conditions. Nondeterministic builds make it impossible to verify that what you tested is what you deployed.
- Pin dependency versions exactly — use lock files and avoid version ranges in production.
- Remove timestamps from build output — use deterministic build tools or strip metadata.
- Seed random generators in tests — use fixed seeds so test failures are reproducible.
- Avoid parallelism in tests that share state — use isolated test databases or in-memory stores.
Dependency Failures
External registries go down. Package mirrors become unavailable. CDN outages break builds worldwide. These failures are outside your control but you can mitigate them with mirrors, local caches, and fallback strategies.
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup npm with fallback registry
run: |
echo "registry=https://registry.npmjs.org/" > .npmrc
echo "registry=https://registry.npmjs.org/ --fallback=true" >> .npmrc
- name: Install with retry
uses: nick-fields/retry@v3
with:
timeout_minutes: 5
max_attempts: 3
command: npm ci --prefer-offline
- run: npm testThe --prefer-offline flag uses locally cached packages when available. The retry action handles transient network failures. Together, they make dependency installation resilient to registry outages.
Debugging Strategies
When pipelines fail, systematic debugging saves time. Start with the failing step, check the logs, reproduce locally, and fix the root cause — not the symptom.
# GitHub Actions debug logging
# Add to repository settings or workflow:
env:
ACTIONS_RUNNER_DEBUG: true
ACTIONS_STEP_DEBUG: true
jobs:
debug:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Debug environment
run: |
echo "Runner OS: $RUNNER_OS"
echo "GitHub ref: $GITHUB_REF"
echo "GitHub SHA: $GITHUB_SHA"
echo "Disk usage:"
df -h
echo "Environment variables:"
env | sort
- name: Setup tmate session
if: failure()
uses: mxschmitt/action-tmate@v3
with:
limit-access-to-actor: trueThe tmate action opens an interactive SSH session when a step fails. You can SSH into the runner and debug interactively. Use limit-access-to-actor to restrict access to the PR author.
Add workflow_dispatch to your workflow triggers so you can run the pipeline manually with custom inputs. This lets you test fixes without pushing a commit, and you can add debug inputs to control verbosity.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.