Stage 7 · Master

Automation Scripts

CSV & Report Generation

Read/write CSV, generate HTML/Markdown reports, and email summaries.

7 min readPython for DevOpsMaster

The csv Module

The csv module in the standard library handles most CSV tasks. Use it for simple reads and writes. Reach for pandas when you need aggregation, filtering, or complex transformations.

Read and write CSV with the csv module


import csv
from pathlib import Path

# Read CSV
def read_server_inventory(path: Path) -> list[dict]:
    with open(path, encoding="utf-8") as f:
        reader = csv.DictReader(f)
        return list(reader)

# Write CSV
def write_status_report(rows: list[dict], path: Path) -> None:
    fieldnames = ["host", "status", "cpu_percent", "memory_percent", "last_check"]
    with open(path, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=fieldnames)
        writer.writeheader()
        writer.writerows(rows)

# Usage
servers = [
    {"host": "web-01", "status": "healthy", "cpu_percent": 45, "memory_percent": 62, "last_check": "2024-01-15T14:30:00"},
    {"host": "web-02", "status": "degraded", "cpu_percent": 89, "memory_percent": 78, "last_check": "2024-01-15T14:30:00"},
]
write_status_report(servers, Path("status_report.csv"))

csv.DictReader maps each row to a dictionary using the header row as keys. csv.DictWriter writes dictionaries as rows using fieldnames as the column order.

DictReader and DictWriter

Filter and transform CSV data


import csv
from pathlib import Path

def filter_unhealthy_servers(input_path: Path, output_path: Path) -> int:
    """Extract servers with high CPU or memory usage."""
    count = 0
    with open(input_path, encoding="utf-8") as fin, \
         open(output_path, "w", newline="", encoding="utf-8") as fout:
        reader = csv.DictReader(fin)
        writer = csv.DictWriter(fout, fieldnames=reader.fieldnames)
        writer.writeheader()

        for row in reader:
            cpu = int(row["cpu_percent"])
            mem = int(row["memory_percent"])
            if cpu > 80 or mem > 80:
                writer.writerow(row)
                count += 1
    return count

unhealthy = filter_unhealthy_servers(
    Path("status_report.csv"),
    Path("unhealthy_servers.csv"),
)
print(f"Found {unhealthy} unhealthy servers")

Streaming row-by-row keeps memory constant regardless of file size. This works for CSV files with millions of rows.

Pandas Basics

pandas is the standard data analysis library in Python. For DevOps reporting, you rarely need the full pandas API — just read, filter, aggregate, and export.

pandas for DevOps reporting


import pandas as pd
from pathlib import Path

# Read CSV into a DataFrame
df = pd.read_csv("status_report.csv")

# Filter rows
unhealthy = df[(df["cpu_percent"] > 80) | (df["memory_percent"] > 80)]

# Aggregate
avg_cpu = df.groupby("status")["cpu_percent"].mean()
print(avg_cpu)

# Summary statistics
print(df.describe())

# Export to CSV
unhealthy.to_csv("unhealthy.csv", index=False)

# Export to Excel (requires openpyxl)
# unhealthy.to_excel("unhealthy.xlsx", index=False)

pandas reads CSV, Excel, JSON, and SQL directly. The DataFrame API is ideal for filtering, grouping, and summarizing operational data.

HTML Report Generation

Generate HTML reports with Jinja2


from jinja2 import Template
from pathlib import Path
import json

TEMPLATE = Template("""
<!DOCTYPE html>
<html>
<head><title>{{ title }}</title></head>
<body>
<h1>{{ title }}</h1>
<p>Generated: {{ timestamp }}</p>
<table border="1" cellpadding="8">
<tr><th>Host</th><th>Status</th><th>CPU %</th><th>Memory %</th></tr>
{% for row in rows %}
<tr class="{{ 'warning' if row.cpu > 80 else '' }}">
  <td>{{ row.host }}</td>
  <td>{{ row.status }}</td>
  <td>{{ row.cpu }}</td>
  <td>{{ row.memory }}</td>
</tr>
{% endfor %}
</table>
</body>
</html>
""")

def generate_html_report(rows: list[dict], title: str) -> str:
    return TEMPLATE.render(title=title, rows=rows, timestamp="2024-01-15")

html = generate_html_report(rows, "Server Health Report")
Path("report.html").write_text(html, encoding="utf-8")

Jinja2 templates keep HTML generation clean. The template defines the structure, Python provides the data. This separates presentation from logic.

Markdown Reports

Generate Markdown reports


from datetime import datetime

def generate_markdown_report(servers: list[dict]) -> str:
    """Generate a Markdown incident report."""
    lines = [
        f"# Server Health Report",
        f"",
        f"**Generated:** {datetime.now().isoformat()}",
        f"**Total servers:** {len(servers)}",
        f"",
        f"## Summary",
        f"",
        f"| Host | Status | CPU % | Memory % |",
        f"|------|--------|-------|----------|",
    ]

    for s in servers:
        status_icon = "+" if s["status"] == "healthy" else "!"
        lines.append(
            f"| {s['host']} | {status_icon} {s['status']} "
            f"| {s['cpu_percent']} | {s['memory_percent']} |"
        )

    unhealthy = [s for s in servers if s["cpu_percent"] > 80]
    if unhealthy:
        lines.extend([
            "",
            "## Unhealthy Servers",
            "",
        ])
        for s in unhealthy:
            lines.append(f"- **{s['host']}**: CPU {s['cpu_percent']}%, Memory {s['memory_percent']}%")

    return "\n".join(lines)

Markdown renders in Slack, GitHub, email (with HTML converter), and most ticketing systems. It is the most portable report format for DevOps.

Emailing Reports

Send reports via email


import smtplib
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart

def send_report(
    to: str,
    subject: str,
    body: str,
    smtp_host: str = "smtp.gmail.com",
    smtp_port: int = 587,
) -> None:
    msg = MIMEMultipart()
    msg["From"] = "ops@example.com"
    msg["To"] = to
    msg["Subject"] = subject

    msg.attach(MIMEText(body, "plain"))

    with smtplib.SMTP(smtp_host, smtp_port) as server:
        server.starttls()
        server.login("ops@example.com", "app-password")
        server.send_message(msg)

# Usage
report = generate_markdown_report(servers)
send_report(
    to="oncall@example.com",
    subject="Daily Server Health Report",
    body=report,
)

For production use, store SMTP credentials in environment variables or a secrets manager. Never hardcode passwords in scripts.

Use pandas for quick one-off analysis

When you need to answer a question like 'which server had the most errors last week', pandas can do it in 3 lines. Do not over-engineer a script when a notebook or REPL session is faster.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.