Stage 7 · Master
LLM Cost, Latency & Observability
Token Budgeting
Measure prompt, completion, retrieval, and tool-call tokens with per-feature budgets and alerts.
7 min readAI for SRE WorkflowsMaster
Why Token Budgets?
Token budgets prevent cost surprises. Without budgets, a single runaway feature can consume your entire monthly LLM budget. Budgets also force you to think about efficiency — every token has a cost.
Token Categories
| Category | What It Includes | Cost Impact |
|---|---|---|
| Prompt tokens | System prompt + context + user query | Lower cost per token |
| Completion tokens | Model output | Higher cost per token |
| Retrieval tokens | Embedding queries + stored chunks | Separate embedding cost |
| Tool-call tokens | Function schemas + tool results | Adds to prompt tokens |
Budget Framework
Token budget manager
class TokenBudgetManager:
def __init__(self):
self.budgets = {
"alert_triage": {
"daily_input_tokens": 500000,
"daily_output_tokens": 100000,
"per_request_max": 4000,
"monthly_cost_limit": 500,
},
"runbook_lookup": {
"daily_input_tokens": 200000,
"daily_output_tokens": 50000,
"per_request_max": 8000,
"monthly_cost_limit": 200,
},
"incident_summary": {
"daily_input_tokens": 1000000,
"daily_output_tokens": 200000,
"per_request_max": 16000,
"monthly_cost_limit": 1000,
},
}
self.usage = defaultdict(lambda: {"input": 0, "output": 0, "cost": 0})
def check_budget(self, feature, estimated_input, estimated_output):
"""Check if a request is within budget."""
budget = self.budgets.get(feature)
if not budget:
return {"allowed": True}
current = self.usage[feature]
if estimated_input > budget["per_request_max"]:
return {"allowed": False, "reason": "Exceeds per-request limit"}
if current["input"] + estimated_input > budget["daily_input_tokens"]:
return {"allowed": False, "reason": "Daily input token limit reached"}
if current["cost"] >= budget["monthly_cost_limit"]:
return {"allowed": False, "reason": "Monthly cost limit reached"}
return {"allowed": True}
def record_usage(self, feature, input_tokens, output_tokens, model):
"""Record token usage and cost."""
cost = calculate_cost(model, input_tokens, output_tokens)
self.usage[feature]["input"] += input_tokens
self.usage[feature]["output"] += output_tokens
self.usage[feature]["cost"] += costBudgets are checked before each request. Usage is recorded after each request. Alerts fire when approaching limits.
Monitoring Usage
Daily budget report
def generate_budget_report(budget_manager):
"""Generate a daily budget usage report."""
report = []
for feature, usage in budget_manager.usage.items():
budget = budget_manager.budgets[feature]
report.append({
"feature": feature,
"input_tokens": usage["input"],
"input_budget_pct": usage["input"] / budget["daily_input_tokens"] * 100,
"output_tokens": usage["output"],
"output_budget_pct": usage["output"] / budget["daily_output_tokens"] * 100,
"cost_usd": usage["cost"],
"cost_budget_pct": usage["cost"] / budget["monthly_cost_limit"] * 100,
})
# Alert on high usage
for r in report:
if r["input_budget_pct"] > 80:
send_alert(f"WARNING: {r['feature']} at {r['input_budget_pct']:.0f}% of daily input budget")
if r["cost_budget_pct"] > 80:
send_alert(f"WARNING: {r['feature']} at {r['cost_budget_pct']:.0f}% of monthly cost budget")
return reportDaily reports show budget consumption. Alerts fire at 80% to give you time to react before hitting limits.
Optimization Techniques
- Cache frequent queries to avoid re-computing embeddings and completions.
- Compress context by summarizing retrieved documents before sending to the model.
- Use smaller models for classification and larger models for complex reasoning.
- Truncate long documents to the most relevant sections.
- Batch multiple queries into a single API call to share overhead.
Set alerts at 60% and 80%
Alert at 60% of daily budget to plan ahead, and at 80% to take immediate action. By 100% it is too late.
Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.