Predictive & Leading Signals

Forecasting disk fill, capacity, and saturation before they page.

7 min readAI for SRE WorkflowsMaster

Reactive vs Predictive

Most monitoring is reactive — alert when a threshold is crossed. Predictive monitoring forecasts when a threshold will be crossed, giving you time to act before the outage.

Approach	When You Know	Response Time
Reactive	After the threshold is crossed	Minutes to hours
Predictive	Before the threshold is crossed	Hours to days

Leading Indicators

Leading indicators are metrics that change before the main metric crosses a threshold. Disk utilization growth rate predicts when disk will be full. Error rate acceleration predicts when error budget will be consumed.

Disk fill rate — bytes written per hour predicts when disk reaches capacity.
Connection pool utilization — increasing utilization predicts connection exhaustion.
Memory growth rate — heap growth predicts OOMKill timing.
Request queue depth — growing queue predicts saturation.
Certificate expiry — days until expiry predicts when to rotate.

Forecasting with ML

Simple linear forecast


import numpy as np
from datetime import datetime, timedelta

def forecast_disk_fill(disk_usage_history, disk_capacity_gb):
    """Forecast when disk will be full using linear regression."""
    # disk_usage_history: list of (timestamp_gb, usage_gb) tuples
    times = np.array([t for t, _ in disk_usage_history])
    usage = np.array([u for _, u in disk_usage_history])

    # Fit linear regression
    coeffs = np.polyfit(times, usage, 1)
    growth_rate_gb_per_hour = coeffs[0]  # GB per hour

    # Current usage and remaining capacity
    current_usage = usage[-1]
    remaining_gb = disk_capacity_gb - current_usage

    # Forecast
    hours_until_full = remaining_gb / growth_rate_gb_per_hour if growth_rate_gb_per_hour > 0 else float('inf')
    estimated_full = datetime.now() + timedelta(hours=hours_until_full)

    return {
        "current_usage_gb": current_usage,
        "growth_rate_gb_per_hour": growth_rate_gb_per_hour,
        "hours_until_full": round(hours_until_full, 1),
        "estimated_full_date": estimated_full.isoformat(),
        "alert_threshold_hours": 24,  # Alert if full within 24 hours
    }

Linear forecasting works well for metrics with steady growth like disk usage. For cyclical metrics, use seasonal models.

Capacity Planning

Capacity forecast report


def generate_capacity_forecast(services):
    """Generate a capacity forecast for all services."""
    forecasts = []
    for service in services:
        metrics = get_usage_metrics(service)
        forecast = {
            "service": service["name"],
            "disk": forecast_disk_fill(metrics["disk"], service["disk_capacity"]),
            "memory": forecast_disk_fill(metrics["memory"], service["memory_capacity"]),
            "cpu": forecast_disk_fill(metrics["cpu"], service["cpu_capacity"]),
        }

        # Flag services at risk
        for resource in ["disk", "memory", "cpu"]:
            if forecast[resource]["hours_until_full"] < 168:  # 7 days
                forecast["at_risk"] = True
                forecast["risk_resource"] = resource

        forecasts.append(forecast)

    return sorted(forecasts, key=lambda x: x.get("disk", {}).get("hours_until_full", 999))

Capacity forecasts give you a weekly view of which services need attention before they run out of resources.

Implementation

Prometheus-based forecasting


import requests

def get_prometheus_forecast(query, hours_ahead=24):
    """Use Prometheus recording rules for forecasting."""
    # Record the growth rate
    recording_rule = f"""
    rate({query}[1h])  # Growth rate per hour
    """
    # Evaluate the rate
    rate_response = requests.get(
        "http://prometheus:9090/api/v1/query",
        params={"query": recording_rule}
    )
    rate = float(rate_response.json()["data"]["result"][0]["value"][1])

    # Get current value
    current_response = requests.get(
        "http://prometheus:9090/api/v1/query",
        params={"query": query}
    )
    current = float(current_response.json()["data"]["result"][0]["value"][1])

    return {
        "current": current,
        "rate_per_hour": rate,
        "forecast": current + rate * hours_ahead,
    }

Use Prometheus recording rules to compute growth rates, then project forward to forecast when thresholds will be crossed.

Predictive monitoring reduces paging by 30 to 50 percent

Teams that implement predictive capacity monitoring report significantly fewer pages. Instead of getting paged at 2 AM when disk fills, they get a ticket during business hours to plan a resize.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.

Stage 7 · Master

AIOps: Detection & Triage