Stage 6 · Operate

Toil Reduction Strategies

Automated Capacity Management

Predictive scaling, quota management, and resource optimization — eliminating manual capacity planning.

7 min readSRE PlaybookOperate

Capacity as Toil

Manual capacity planning is pure toil. It scales linearly with services, is repetitive, and can be automated. Every time a human checks whether a service has enough resources and manually scales it, that is toil. Capacity automation eliminates this by making scaling reactive, predictive, or both.

Reactive vs predictive scaling

Reactive scaling responds to current load — it scales up after traffic increases. Predictive scaling anticipates future load — it scales up before traffic increases. Both reduce toil, but predictive scaling prevents user impact.

Predictive Scaling

Predictive scaling uses historical traffic patterns to anticipate demand. If your service peaks at 9 AM every weekday, predictive scaling adds capacity at 8:45 AM. This eliminates the lag between traffic increase and scale-up.

Kubernetes predictive scaling with KEDA


apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: user-api-predictive
  namespace: production
spec:
  scaleTargetRef:
    name: user-api
  minReplicaCount: 3
  maxReplicaCount: 20
  triggers:
    # Predictive scaling based on cron patterns
    - type: cron
      metadata:
        timezone: America/New_York
        start: "0 8 * * 1-5"    # Weekdays 8 AM
        end: "0 18 * * 1-5"     # Weekdays 6 PM
        desiredReplicas: "8"

    # Reactive scaling based on CPU
    - type: cpu
      metricType: Utilization
      metadata:
        value: "70"

    # Reactive scaling based on queue depth
    - type: rabbitmq
      metadata:
        queueName: user-api-requests
        queueLength: "100"

Quota Automation

Resource quotas prevent runaway scaling from consuming all cluster resources. Automate quota management to adjust limits based on actual usage patterns. This eliminates the manual process of requesting and approving quota increases.

Automated quota adjustment controller


apiVersion: v1
kind: ConfigMap
metadata:
  name: quota-controller-config
  namespace: kube-system
data:
  config.yaml: |
    namespaces:
      - name: production
        check_interval: 1h
        thresholds:
          cpu_request_ratio: 0.8     # Scale up quota if usage > 80%
          memory_request_ratio: 0.85
        actions:
          increase:
            cpu: "4"                 # Add 4 CPU cores
            memory: "8Gi"            # Add 8Gi memory
          max_quota:
            cpu: "32"
            memory: "64Gi"
        notifications:
          channel: "#capacity-alerts"
          on_increase: true
          on_approaching_limit: true

Resource Optimization

Automated resource optimization rightsizes your workloads. It analyzes actual resource usage and adjusts requests and limits to match. This reduces costs without sacrificing reliability.

Resource optimization with VPA


apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: user-api-vpa
  namespace: production
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: user-api
  updatePolicy:
    updateMode: "Off"  # Only recommend, do not auto-apply
  resourcePolicy:
    containerPolicies:
      - containerName: user-api
        minAllowed:
          cpu: 50m
          memory: 64Mi
        maxAllowed:
          cpu: 2
          memory: 2Gi
        controlledResources: ["cpu", "memory"]

Start with recommendations

Start VPA in recommendation mode (updateMode: Off). Review the recommendations manually before enabling auto-apply. This prevents VPA from making aggressive changes that could destabilize your service.

Kubernetes HPA

The Horizontal Pod Autoscaler is the most common capacity automation tool in Kubernetes. It scales pod count based on CPU, memory, or custom metrics. Configure it with appropriate targets, cooldowns, and behavior policies.

HPA with behavior policies


apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: user-api-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: user-api
  minReplicas: 3
  maxReplicas: 15
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
        - type: Percent
          value: 100
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 10
          periodSeconds: 60

Capacity Dashboards

Automated capacity management needs visibility. Build dashboards that show current utilization, predicted growth, quota usage, and cost impact. This gives leadership confidence that automation is working.

Capacity metrics in Prometheus


groups:
  - name: capacity_metrics
    rules:
      # Cluster utilization
      - record: cluster:cpu:utilization:ratio
        expr: |
          sum(kube_pod_container_resource_requests{resource="cpu"})
          /
          sum(kube_node_status_allocatable{resource="cpu"})

      # Quota usage
      - record: namespace:quota:cpu:usage:ratio
        expr: |
          sum(kube_resourcequota{resource="cpu", type="used"})
          /
          clamp_min(sum(kube_resourcequota{resource="cpu", type="hard"}), 1)

      # Cost per request (estimated)
      - record: service:cost:per_request:daily
        expr: |
          sum(service:resource:cost:daily)
          /
          clamp_min(sum(rate(http_request_duration_seconds_count[24h])), 1)

Capacity is a cost conversation

Capacity automation is not just about reliability — it is about cost. Over-provisioning wastes money. Under-provisioning causes outages. Automated capacity management finds the sweet spot and keeps you there.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.