Stage 7 · Master
ChatOps & On-Call Copilots
The On-Call Copilot
Answering what does this alert mean with runbook-grounded context.
Enriching Alerts with Context
When an alert fires, the on-call engineer needs to quickly understand what it means, what service is affected, and what to do. The copilot enriches raw alert data with runbook context, recent changes, and historical incidents.
def enrich_alert(alert):
"""Enrich a raw alert with contextual information."""
# 1. Retrieve relevant runbook sections
runbook_chunks = hybrid_retrieve(
query=alert["description"],
filters={"service": alert["service"]},
top_k=3
)
# 2. Find similar past incidents
past_incidents = search_incidents(
query=alert["description"],
service=alert["service"],
top_k=3
)
# 3. Get recent changes
recent_changes = get_recent_changes(
service=alert["service"],
hours=24
)
return {
"alert": alert,
"runbook": runbook_chunks,
"similar_incidents": past_incidents,
"recent_changes": recent_changes,
}Context enrichment gives the model everything it needs to provide a useful, grounded answer.
Runbook-Grounded Answers
COPILOT_SYSTEM = """You are an on-call assistant for a cloud infrastructure team.
Given an alert and the relevant context, provide:
1. What this alert means in plain English
2. What service is affected
3. The recommended first steps from the runbook
4. Whether this is likely related to recent changes
5. Similar past incidents and their resolution
RULES:
- Only use information from the provided runbook and incident context
- Cite specific runbook sections by name
- If the runbook does not cover this alert, say so explicitly
- Never guess about commands or procedures not in the runbook"""
def oncall_answer(enriched_alert):
context = format_enriched_context(enriched_alert)
response = openai.ChatCompletion.create(
model="gpt-4",
temperature=0.0,
messages=[
{"role": "system", "content": COPILOT_SYSTEM},
{"role": "user", "content": context}
]
)
return response["choices"][0]["message"]["content"]The system prompt explicitly tells the model to cite runbooks and avoid guessing. This grounds every answer in your actual procedures.
Conversation Memory
On-call investigations are iterative. The engineer asks follow-up questions. The copilot needs to remember what was discussed and build on previous answers.
from collections import defaultdict
import time
class ConversationStore:
def __init__(self, ttl_hours=4):
self.conversations = defaultdict(list)
self.ttl = ttl_hours * 3600
def add_message(self, channel_id, role, content):
self.conversations[channel_id].append({
"role": role,
"content": content,
"timestamp": time.time()
})
def get_messages(self, channel_id):
"""Get recent messages within TTL."""
now = time.time()
messages = self.conversations[channel_id]
# Filter expired messages
messages = [m for m in messages if now - m["timestamp"] < self.ttl]
self.conversations[channel_id] = messages
return [{"role": m["role"], "content": m["content"]} for m in messages]
def clear(self, channel_id):
self.conversations.pop(channel_id, None)
# Usage
store = ConversationStore(ttl_hours=4)
def oncall_conversation(channel_id, user_message):
store.add_message(channel_id, "user", user_message)
messages = [
{"role": "system", "content": COPILOT_SYSTEM},
*store.get_messages(channel_id)
]
response = openai.ChatCompletion.create(
model="gpt-4",
messages=messages
)
assistant_msg = response["choices"][0]["message"]["content"]
store.add_message(channel_id, "assistant", assistant_msg)
return assistant_msgConversation memory lets the copilot maintain context across follow-up questions without re-sending the entire history.
Building the Copilot
- Alert arrives via PagerDuty webhook or Prometheus Alertmanager.
- Copilot enriches the alert with runbook chunks, past incidents, and recent changes.
- Enriched context is sent to the LLM with the on-call system prompt.
- Response is posted to the incident Slack channel.
- Engineer can ask follow-up questions in the thread.
- Copilot remembers conversation context for the session.
Feedback Loop
Track which answers engineers found helpful. Use reactions (thumbs up, thumbs down) as implicit feedback. This data improves your RAG system over time.
@app.event("reaction_added")
def handle_reaction(event, say):
"""Track user feedback on bot responses."""
if event["reaction"] == "+1":
log_feedback(event["item"]["ts"], positive=True)
elif event["reaction"] == "-1":
log_feedback(event["item"]["ts"], positive=False)
say(
text="Thanks for the feedback. What was wrong with the answer?",
thread_ts=event["item"]["ts"]
)Thumbs down feedback triggers a follow-up question that helps you understand what went wrong and improve the system.
Track how quickly the copilot provides a useful answer after an alert fires. The goal is under 30 seconds from alert to actionable guidance.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.