Stage 4 · Provision

Design Case Studies

Design an Audit Log

Append-only storage, tamper evidence, retention policies, and queryable compliance exports.

7 min readSystem Design for SREProvision

Requirements

An audit log records every significant action performed in the system. It must be append-only (no modifications or deletions), tamper-evident (detect alterations), retainable for years, and queryable for compliance audits. Audit logs are legal requirements in many industries.

Append-Only Storage

Audit log entry


audit_entry:
  id: "evt_abc123"
  timestamp: "2025-01-15T10:30:45.123Z"
  actor:
    id: "user_456"
    email: "alice@example.com"
    ip: "203.0.113.42"
    user_agent: "Mozilla/5.0..."
  action: "DELETE"
  resource:
    type: "database_user"
    id: "db_user_789"
    name: "readonly-replica"
  details:
    reason: "Employee offboarding"
    ticket_id: "INC-1234"
  result: "success"
  checksum: "sha256:a1b2c3..."

Every audit entry records who did what, when, where, and the result. The checksum enables tamper detection. The ticket_id provides an audit trail reference.

Tamper Evidence

Tamper evidence ensures that audit log entries cannot be modified or deleted without detection. Use hash chains: each entry includes the hash of the previous entry. Modifying any entry breaks the chain, making tampering detectable.

Hash chain implementation


import hashlib
import json

def create_audit_entry(entry: dict, previous_hash: str) -> dict:
    entry['previous_hash'] = previous_hash
    entry_json = json.dumps(entry, sort_keys=True)
    entry['checksum'] = hashlib.sha256(entry_json.encode()).hexdigest()
    return entry

# Chain of entries:
# Entry 1: previous_hash = "0", checksum = hash(entry1)
# Entry 2: previous_hash = hash(entry1), checksum = hash(entry2)
# Entry 3: previous_hash = hash(entry2), checksum = hash(entry3)

# Verify chain integrity:
def verify_chain(entries: list) -> bool:
    for i in range(1, len(entries)):
        if entries[i]['previous_hash'] != entries[i-1]['checksum']:
            return False  # Chain broken — tampering detected
    return True

Each entry includes the hash of the previous entry, forming a chain. Any modification to an entry breaks the chain. This is the same principle used in blockchains.

Retention Policies

Regulation	Retention	Requirement
SOC 2	1 year minimum	Access logs, configuration changes
HIPAA	6 years	Access to PHI, modifications
PCI DSS	1 year	Access to cardholder data
GDPR	Varies	Processing activities, consent
SOX	7 years	Financial data access

Retention policy configuration


retention_policies:
  - category: "security_events"
    retention: "7_years"
    storage: "S3 Glacier Deep Archive"
    compliance: ["SOC2", "HIPAA", "PCI_DSS"]
  
  - category: "access_logs"
    retention: "3_years"
    storage: "S3 Standard"
    compliance: ["SOC2"]
  
  - category: "configuration_changes"
    retention: "5_years"
    storage: "S3 Standard → Glacier after 90 days"
    compliance: ["SOC2", "SOX"]

Different categories of audit events have different retention requirements. Store hot data in S3 Standard and transition to Glacier for long-term retention.

Queryable Compliance Exports

Compliance auditors need to query audit logs by actor, resource, action, and time range. Use a searchable index (Elasticsearch) for interactive queries and S3 for long-term archival. Export filtered logs as CSV or PDF for auditors.

Audit log query examples


Compliance auditor queries:
  - Show all access to database X in the last 90 days
  - List all configuration changes by user Y
  - Show all failed login attempts in the last 30 days
  - Export all audit entries for resource Z as CSV

Implementation:
  Elasticsearch index: audit-logs-2025-01
  Fields: actor.id, resource.type, action, timestamp
  Query: actor.id:"user_456" AND action:"DELETE" AND timestamp:[2025-01-01 TO 2025-01-31]

Elasticsearch provides fast, flexible querying for compliance audits. S3 stores the complete archive. Elasticsearch is rebuilt from S3 for disaster recovery.

Audit Log Architecture

Audit log pipeline


Application (any action)
  │
  ▼
Audit Event (Kafka)
  │
  ├──► Elasticsearch (queryable index, 90 days)
  │
  ├──► S3 (long-term archive, 7 years)
  │    └── Lifecycle: Standard → IA → Glacier → Deep Archive
  │
  └──► Alerting (sensitive actions trigger alerts)
       ├── Root user actions
       ├── Configuration changes
       └── Failed authentication attempts

The audit pipeline writes to multiple destinations: Elasticsearch for querying, S3 for archival, and alerting for sensitive actions. Each destination serves a different purpose.

Audit logs are immutable by design

Never allow modification or deletion of audit log entries. Use append-only storage, hash chains, and access controls. Audit logs that can be tampered with are worse than no audit logs at all.

Mark this lesson complete to store local progress and unlock a cleaner resume path the next time you visit.