Inhibition and Silences

Inhibition Rules

Inhibition rules suppress alerts when a more severe alert is already firing. When a critical alert fires, related warning alerts become noise. Inhibition rules automatically silence the less severe alerts, reducing notification fatigue.

inhibit_rules:
  # Suppress all warnings when critical is firing
  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal: ["job"]

  # Suppress service alerts when node is down
  - source_match:
      alertname: NodeDown
    target_match_re:
      alertname: ".+High.+|.+Error.+"
    equal: ["instance"]

Inhibition Example

Consider a scenario where a node goes down. You have both a NodeDown critical alert and a HighCPU warning alert for the same node. Without inhibition, both alerts fire. With inhibition, the HighCPU alert is suppressed because the node is already known to be down.

Silences

Silences temporarily suppress notifications for alerts matching specific label matchers. They are used during maintenance windows, known outages, or when you need to suppress false positives temporarily. Silences do not prevent alerts from firing, only from being sent.

# Silence a specific alert for 4 hours
amtool silence add alertname=HighCPU instance=web-01 \
  --duration=4h \
  --comment="Planned maintenance on web-01"

# Silence all alerts for a job
amtool silence add job=database \
  --duration=2h \
  --comment="Database migration in progress"

Silence Syntax

Silences use the same label matchers as routes. They support exact match, regex match, and negative match. A silence expires automatically after its duration or when manually expired.

curl -X POST http://alertmanager:9093/api/v2/silences -d '{
  "matchers": [
    {"name": "alertname", "value": "HighLatency", "isRegex": false},
    {"name": "job", "value": "api", "isRegex": false}
  ],
  "startsAt": "2024-01-15T02:00:00Z",
  "endsAt": "2024-01-15T06:00:00Z",
  "createdBy": "oncall-engineer",
  "comment": "Planned API deployment"
}'

Maintenance Windows

• Always create silences before starting maintenance, not after alerts fire.
• Set the minimum duration needed. Longer silences risk hiding real incidents.
• Include the reason in the comment field. Future on-call engineers need context.
• Use regex matchers to silence all alerts for a service or environment.
• Expire silences immediately after maintenance completes.