Stage 6 · Operate

Alerting Hygiene & Paging Design

Noise Reduction

Deduplication, grouping, inhibition rules, flapping detection, and stale alert cleanup.

7 min readSRE PlaybookOperate

The Noise Problem

Alert noise is the single biggest threat to on-call sustainability. When engineers receive 50 pages per day, they stop paying attention. Noise causes fatigue, which causes missed real incidents. Reducing noise is not optional.

Noise kills reliability

Alert noise directly causes missed real incidents. An engineer who receives 50 noisy pages per day will eventually ignore the 51st page — which might be the one that matters.

Deduplication

Deduplication prevents the same alert from firing multiple times. Alertmanager groups alerts by labels and sends grouped notifications. This prevents alert storms where one issue triggers dozens of pages.

Alertmanager deduplication configuration


deduplication:
  strategy: "group_by"
  group_by_labels:
    - "alertname"
    - "service"
    - "team"

  group_wait: "30s"
  group_interval: "5m"
  repeat_interval: "4h"

  # Example: 10 pods with high CPU
  # Without dedup: 10 separate pages
  # With dedup: 1 grouped page with all 10 pods listed

  example:
    alerts:
      - alertname: HighCPU
        pod: "user-api-abc123"
        service: user-api
      - alertname: HighCPU
        pod: "user-api-def456"
        service: user-api
      - alertname: HighCPU
        pod: "user-api-ghi789"
        service: user-api

    grouped_notification:
      title: "HighCPU (3 alerts)"
      body: |
        - user-api-abc123: CPU 95%
        - user-api-def456: CPU 92%
        - user-api-ghi789: CPU 88%

Alert Grouping

Group related alerts into single notifications. Instead of 10 separate disk space alerts, group them into one notification listing all affected nodes. This reduces noise while preserving information.

Alert grouping strategies


grouping_strategies:
  by_service:
    labels: ["service"]
    effect: "All alerts for a service grouped together"
    best_for: "Small services with few alerts"

  by_cluster:
    labels: ["cluster"]
    effect: "All alerts for a cluster grouped together"
    best_for: "Multi-cluster environments"

  by_team:
    labels: ["team"]
    effect: "All alerts for a team grouped together"
    best_for: "Large organizations"

  by_alert_and_service:
    labels: ["alertname", "service"]
    effect: "Same alert type for same service grouped"
    best_for: "Most environments"

  custom:
    labels: ["alertname", "service", "cluster"]
    effect: "Highly specific grouping"
    best_for: "Complex microservices"

Inhibition Rules

Inhibition rules suppress lower-severity alerts when higher-severity alerts are firing. If a service is completely down, you do not need a separate page for high CPU. The critical alert covers it.

Alertmanager inhibition rules


inhibit_rules:
  # Inhibit warnings if critical is firing for same service
  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal: ["service"]

  # Inhibit info if warning is firing for same service
  - source_match:
      severity: warning
    target_match:
      severity: info
    equal: ["service"]

  # Inhibit individual pod alerts if service-wide alert is firing
  - source_match:
      alertname: ServiceDown
    target_match:
      alertname: HighCPU
    equal: ["service"]

  # Inhibit CPU alerts if service is being scaled
  - source_match:
      alertname: AutoscalingActive
    target_match:
      alertname: HighCPU
    equal: ["service"]

  example:
    scenario: "Service returns 503 for all requests"
    alerts_firing:
      - "ServiceDown (critical)"
      - "HighErrorRate (critical)"
      - "HighLatency (warning)"
      - "HighCPU (warning)"
      - "DiskSpaceLow (info)"

    alerts_delivered:
      - "ServiceDown (critical)"
      - "HighErrorRate (critical)"

    alerts_inhibited:
      - "HighLatency (inhibited by ServiceDown)"
      - "HighCPU (inhibited by ServiceDown)"
      - "DiskSpaceLow (inhibited by ServiceDown)"

Flapping Detection

Flapping alerts fire, resolve, and fire again repeatedly. This creates noise and trains engineers to ignore alerts. Detect and suppress flapping alerts automatically.

Flapping detection and suppression


flapping_detection:
  definition: "An alert that fires and resolves 3+ times in 1 hour"

  detection:
    metric: |
      count(ALERTS{alertstate="firing"}[1h]) by (alertname, service)
    threshold: 3
    window: "1 hour"

  suppression:
    method: "Increase alert duration"
    original_duration: "5m"
    flapping_duration: "15m"

  recording_rule: |
    # Count alert state changes
    - record: alert:flapping:count
      expr: |
        count(ALERTS{alertstate="firing"}[1h]) by (alertname, service)

    # Identify flapping alerts
    - record: alert:flapping:detected
      expr: |
        alert:flapping:count > 3

    # Suppress flapping alerts with longer duration
    - alert: FlappingAlertSuppressed
      expr: alert:flapping:detected
      for: 15m
      labels:
        severity: info
      annotations:
        summary: "Alert {{ $labels.alertname }} is flapping"
        description: "This alert has fired {{ $value }} times in the last hour"

Stale Alert Cleanup

Stale alerts are alerts that fire but never resolve. They accumulate in your alerting system and create noise. Automatically resolve alerts that have not fired for a configurable period.

Stale alert cleanup configuration


stale_cleanup:
  definition: "An alert that has been firing for > 24 hours without resolution"

  detection:
    metric: |
      ALERTS{alertstate="firing"}
    threshold: "24 hours"
    action: "Notify team, investigate why not resolved"

  automatic_resolution:
    enabled: true
    timeout: "48 hours"
    action: "Auto-resolve and create investigation ticket"

  recording_rule: |
    # Find alerts firing for more than 24 hours
    - record: alert:stale:detected
      expr: |
        ALERTS{alertstate="firing"} and on(alertname, service)
        (time() - ALERTS{alertstate="firing"} start) > 86400

    # Alert on stale alerts
    - alert: StaleAlertDetected
      expr: alert:stale:detected
      for: 1h
      labels:
        severity: warning
      annotations:
        summary: "Alert {{ $labels.alertname }} has been firing for > 24 hours"
        description: "This alert may be stale and should be investigated or resolved"

Noise reduction is continuous

Alert noise reduction is not a one-time project. It is an ongoing practice. Review alert quality weekly, tune noisy alerts monthly, and audit your entire alerting strategy quarterly.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.