Stage 4 · Provision
Observability-Driven Remediation
Post-Remediation Validation
Verify fixes with assert tasks, uri checks, service facts, Prometheus queries, and rollback triggers.
Validation Strategy
A remediation is not complete until you verify it worked. Post-remediation validation confirms the fix by checking service state, running health checks, querying monitoring systems, and triggering rollback if validation fails.
---
- name: Post-remediation validation
hosts: "{{ target_hosts }}"
tasks:
# 1. Check service is running
# 2. Verify health endpoint
# 3. Check logs for errors
# 4. Query Prometheus for alert status
# 5. Verify configuration is correct
# 6. Trigger rollback if any check failsValidate from multiple angles — service state, health endpoints, monitoring data, and configuration. A single check is not enough.
Assert Tasks
- name: Validate remediation
hosts: "{{ target_hosts }}"
tasks:
- name: Check service is running
ansible.builtin.systemd:
name: "{{ service_name }}"
register: service_status
- name: Assert service is active
ansible.builtin.assert:
that:
- service_status.status.ActiveState == "active"
fail_msg: "Service is not active: {{ service_status.status.ActiveState }}"
success_msg: "Service is running correctly"
- name: Check process count
ansible.builtin.command: pgrep -c "{{ service_name }}"
register: process_count
changed_when: false
- name: Assert process is running
ansible.builtin.assert:
that:
- process_count.stdout | int > 0
fail_msg: "No {{ service_name }} processes found"Assert tasks validate conditions and fail with descriptive messages. They are the building blocks of post-remediation validation.
HTTP Health Checks
- name: HTTP health validation
hosts: "{{ target_hosts }}"
tasks:
- name: Wait for service to start
ansible.builtin.wait_for:
port: "{{ service_port }}"
timeout: 60
- name: Check health endpoint
ansible.builtin.uri:
url: "http://{{ inventory_hostname }}:{{ service_port }}/health"
status_code: 200
return_content: yes
register: health_response
- name: Validate health response
ansible.builtin.assert:
that:
- health_response.status == 200
- health_response.json.status == "healthy"
- health_response.json.version == deploy_version
fail_msg: "Health check failed: {{ health_response.content }}"
- name: Check readiness endpoint
ansible.builtin.uri:
url: "http://{{ inventory_hostname }}:{{ service_port }}/ready"
status_code: 200
retries: 5
delay: 10Health checks verify the application is not just running but actually functional. Check status codes, response bodies, and version numbers.
Service Facts
- name: Gather service facts
ansible.builtin.service_facts:
- name: Validate required services
ansible.builtin.assert:
that:
- "'nginx.service' in ansible_facts.services"
- "ansible_facts.services['nginx.service'].state == 'running'"
- "'postgresql.service' in ansible_facts.services"
- "ansible_facts.services['postgresql.service'].state == 'running'"
fail_msg: "Required services are not running"
- name: Check listening ports
ansible.builtin.wait_for:
port: "{{ item }}"
timeout: 5
loop:
- 80
- 443
- 5432Service facts provide a complete picture of all services on the host. Use them to verify all required services are running.
Prometheus Validation
- name: Query Prometheus for alert status
ansible.builtin.uri:
url: "http://prometheus:9090/api/v1/query"
method: GET
body_format: json
body:
query: "ALERTS{alertname='ServiceDown',instance='{{ inventory_hostname }}'}"
return_content: yes
register: prometheus_result
- name: Verify alert is resolved
ansible.builtin.assert:
that:
- prometheus_result.json.data.result | length == 0
fail_msg: "ServiceDown alert is still firing"
- name: Query error rate
ansible.builtin.uri:
url: "http://prometheus:9090/api/v1/query"
method: GET
body_format: json
body:
query: "rate(http_requests_total{status='5xx',instance='{{ inventory_hostname }}'}[5m])"
return_content: yes
register: error_rate
- name: Assert error rate is acceptable
ansible.builtin.assert:
that:
- error_rate.json.data.result[0].value[1] | float < 0.01
fail_msg: "Error rate is too high: {{ error_rate.json.data.result }}"Query Prometheus to verify the alert is resolved and error rates are acceptable. This confirms the fix actually worked.
A service restart might appear to fix the problem, but if the health check fails or the alert is still firing, the fix did not work. Multi-layered validation catches false positives.
Package post-remediation validation as a reusable playbook. Every remediation playbook should call the validation playbook as its final step. If validation fails, trigger rollback automatically.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.