Stage 7 · Master
AIOps: Detection & Triage
Anomaly Detection
Seasonality, baselines, and catching drift metrics thresholds miss.
Why ML for Anomaly Detection?
Static thresholds fail on dynamic systems. A CPU usage threshold of 80% causes false alarms during peak hours and misses anomalies during off-peak. ML-based anomaly detection learns the normal pattern for each metric and flags deviations from that pattern.
Establishing Baselines
A baseline is the expected behavior of a metric over time. It captures the normal range, including time-of-day and day-of-week patterns. An anomaly is any observation that deviates significantly from the baseline.
import pandas as pd
import numpy as np
def compute_baseline(metric_series, window_hours=168):
"""Compute rolling mean and std for a metric (168 = 1 week)."""
df = pd.DataFrame({"value": metric_series})
df["rolling_mean"] = df["value"].rolling(window=window_hours, min_periods=24).mean()
df["rolling_std"] = df["value"].rolling(window=window_hours, min_periods=24).std()
df["upper_bound"] = df["rolling_mean"] + 3 * df["rolling_std"]
df["lower_bound"] = df["rolling_mean"] - 3 * df["rolling_std"]
return df
def detect_anomalies(df):
"""Flag points outside the baseline bounds."""
df["is_anomaly"] = (df["value"] > df["upper_bound"]) | (df["value"] < df["lower_bound"])
return df[df["is_anomaly"]]A 3-sigma bound works for many metrics. Points outside 3 standard deviations from the rolling mean are flagged as anomalies.
Handling Seasonality
Most infrastructure metrics have strong daily and weekly patterns. Traffic peaks during business hours. Backups run at night. Batch jobs run on weekends. The detection model must account for these patterns.
from statsmodels.tsa.seasonal import seasonal_decompose
def decompose_metric(metric_values, period=24):
"""Decompose a metric into trend, seasonal, and residual."""
result = seasonal_decompose(
metric_values,
model="additive",
period=period # 24 for hourly data with daily seasonality
)
return {
"trend": result.trend,
"seasonal": result.seasonal,
"residual": result.resid, # Anomalies show up here
}
# Detect anomalies in the residual component
# This removes trend and seasonality, leaving only unexpected deviationsAnomalies in the residual component are true anomalies, not just seasonal peaks.
Detection Methods
| Method | Strengths | Weaknesses |
|---|---|---|
| Z-score / 3-sigma | Simple, fast | Assumes normal distribution |
| IQR | Robust to outliers | Misses gradual drift |
| Isolation Forest | Works on multivariate | Needs training data |
| Prophet | Handles seasonality well | Slower, more complex |
| LLM-based | Can reason about context | Expensive, higher latency |
Implementation
from sklearn.ensemble import IsolationForest
import numpy as np
# Features: [cpu_usage, memory_usage, request_rate, error_rate]
X_train = np.array([...]) # Historical normal data
# Train the model
model = IsolationForest(
n_estimators=100,
contamination=0.01, # Expect 1% anomalies
random_state=42
)
model.fit(X_train)
# Detect anomalies in real-time
def detect_anomaly(cpu, memory, request_rate, error_rate):
features = np.array([[cpu, memory, request_rate, error_rate]])
prediction = model.predict(features) # -1 = anomaly, 1 = normal
score = model.decision_function(features) # Lower = more anomalous
return {
"is_anomaly": prediction[0] == -1,
"anomaly_score": float(score[0]),
}Isolation Forest works well for multivariate anomaly detection. It learns which combinations of metrics are normal and flags unusual combinations.
Begin with simple per-metric anomaly detection. Once you have reliable per-metric detection, combine signals with multivariate methods for better accuracy.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.