Stage 4 · Provision

Observability-Driven Remediation

Alert to Playbook

Trigger remediation from Prometheus Alertmanager, webhooks, and Event-Driven Ansible rulebooks.

7 min readAnsible AutomationProvision

The Alert Pipeline

An alert-to-playbook pipeline connects monitoring systems to Ansible automation. When an alert fires, it triggers a webhook that runs an Ansible playbook to diagnose and remediate the issue automatically.

Alert pipeline flow


# 1. Monitoring detects issue (Prometheus, CloudWatch, etc.)
# 2. Alert fires to Alertmanager
# 3. Alertmanager sends webhook to EDA or AWX
# 4. EDA/AWX matches alert to playbook
# 5. Playbook executes remediation
# 6. Results reported back to monitoring

This pipeline converts alerts into automated actions. The time from detection to remediation drops from minutes to seconds.

Alertmanager Webhooks

Alertmanager config


# alertmanager.yml
route:
  receiver: ansible-eda
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

receivers:
  - name: ansible-eda
    webhook_configs:
      - url: "http://eda-server:5000/alerts"
        send_resolved: true

inhibit_rules:
  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal: [instance]

Alertmanager forwards alerts to EDA via webhook. EDA matches the alert to a rulebook and triggers remediation.

EDA Integration

EDA rulebook for Prometheus alerts


---
- name: Handle Prometheus alerts
  hosts: localhost
  sources:
    - name: alertmanager_webhook
      webhook:
        host: 0.0.0.0
        port: 5000
        payload_events:
          - key: status
            value: firing

  rules:
    - name: Restart failed service
      condition: |
        event.alerts[0].labels.alertname == "ServiceDown"
      action:
        run_playbook:
          name: playbooks/restart-service.yml
          extra_vars:
            service_name: "{{ event.alerts[0].labels.service }}"
            target_host: "{{ event.alerts[0].labels.instance }}"
            alert_id: "{{ event.alerts[0].fingerprint }}"

    - name: Disk space critical
      condition: |
        event.alerts[0].labels.alertname == "DiskSpaceCritical"
      action:
        run_playbook:
          name: playbooks/cleanup-disk.yml
          extra_vars:
            target_host: "{{ event.alerts[0].labels.instance }}"
            threshold: "{{ event.alerts[0].annotations.value }}"

EDA rulebooks match specific alert patterns and trigger appropriate playbooks. Each alert type gets its own remediation.

Webhook Receiver Setup

AWX webhook integration


---
- name: Configure AWX webhook receiver
  hosts: localhost
  connection: local
  tasks:
    - name: Create webhook credential
      ansible.controller.credential:
        name: "Webhook Token"
        credential_type: "Token"
        inputs:
          token: "{{ awx_api_token }}"
        state: present
        controller_host: "https://awx.example.com"
        controller_username: "admin"
        controller_password: "{{ awx_password }}"

    - name: Create webhook-triggered job template
      ansible.controller.job_template:
        name: "Remediate Service Down"
        project: "Remediation Playbooks"
        playbook: "playbooks/restart-service.yml"
        inventory: "Production"
        credential: "SSH Deploy Key"
        webhook_service: "github"
        webhook_credential: "Webhook Token"
        state: present
        controller_host: "https://awx.example.com"
        controller_username: "admin"
        controller_password: "{{ awx_password }}"

AWX has built-in webhook support. Configure a job template to accept webhooks from Alertmanager or other monitoring tools.

Alert-to-Playbook Mapping

Alert mapping table


# Alert Name              -> Playbook
# ServiceDown              -> restart-service.yml
# DiskSpaceCritical       -> cleanup-disk.yml
# HighMemoryUsage         -> restart-gc.yml
# SSLCertExpiring         -> renew-cert.yml
# UnauthorizedAccess      -> block-ip.yml
# HighCPUUsage            -> scale-instance.yml
# DatabaseConnectionLost  -> restart-db.yml
# DeployFailed            -> rollback.yml

Document your alert-to-playbook mapping. This is your remediation runbook in code.

Alert-to-playbook is the foundation

Every self-healing system starts with mapping alerts to remediation playbooks. Start with simple, well-understood problems like service restarts and expand from there.

Start with notifications

Before automating remediation, use the same pipeline to send notifications. Verify the mapping is correct and the right playbooks trigger before enabling automatic execution.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.